SAP Datasphere – integration of data from various sources

When connecting SAP Datasphere to S/4HANA or SAP ECC systems, data can be taken over in several ways, using CDS tables and views. They can be loaded into SAP Datasphere or used in remote mode, especially when the data volume is smaller.

This raises the question: if the data is loaded into SAP Datasphere, is delta also supported? Can standard SAPI extractors be used, and if so, to what extent? What does the process of data feeding from external databases or source systems based on third-party cloud environments look like in SAP Datasphere?

In this article, we answer these questions, especially in the context of SAP systems. We present the possibilities of integrating SAP systems with a particular focus on S/4HANA and SAP ECC. We discuss how we handled SAPI extractors in SAP Datasphere for one of our customers. We describe the possibilities of integrating SAP Datasphere with external SQL databases, as well as cloud solutions, with a focus on Google products.

The integration of SAP ECC systems with SAP Datasphere largely depends on the version of SAP you have, as well as the database on which the transaction system is based. Within the test environment, the oldest SAP system we were able to connect to SAP Datasphere was SAP ECC 6.0 in ABAP and BASIS version 7.0. on a SQL database. For this version of the system, the supported data sources were tables available in remote mode with the possibility of saving them in the cloud data warehouse.

Much more opportunities for integration are provided by SAP ECC 6.0 systems based on the HANA database, with the minimum 7.4. version of BASIS and ABAO components. Then, in addition to tables, CDS views can also be the data source. However, in this version of ECC systems, CDS views will not support delta. Furthermore, it is worth noting that in the case of a newer version of the system (BASIS 7.5 component), we managed to operate in the test environment an SLT server allowing for the replication of tables in real time (every few tens of seconds). SAPI extractors are also available, but are not recommended.

It is worth noting that for SAP HANA-based systems, it is possible to connect only the database. Then the source in SAP Datasphere can be defined calculation views.

The SAPI extractors commonly used in SAP ECC systems to export data externally can also be used in SAP Datasphere, but this is not recommended. In the first place, it is not recommended to access them from SAP Datasphere in remote mode. This is because very often extractors are based on complex functional modules, which can significantly affect the performance of data extraction into the data warehouse.

On the other hand, when data is loaded into SAP Datasphere through data flow from extractors, the operation of the delta mechanism is hindered. For it to work effectively, the key fields of an extractor should be identical to the key fields of an internal table on the SAP Datasphere side. However, many standard extractors do not have such key fields defined, and it is not recommended to create them. So is it impossible to use SAPI extractors at all for SAP Datasphere?

For one of our customers, it was important to enable data extraction into SAP Datasphere using 2lis* extractors, which had been already widely used, tested and extended. Since it was not possible to support them as a standard, the problem was solved by using the SAP BW component built into SAP ECC. From a technical point of view, the solution required the following steps on the SAP ECC side in the RSA1 transaction:

• Replication and activation of the data sources we wanted to use in SAP Datasphere;
• Creation of infopackages that allowed us to support the standard delta mechanism;
• Determining the technical name of the so-called PSA table, which was to receive data from the loaded deltas;
• Implementing a data loading chain using infopackages into the PSA table and cleaning it regularly (every 30 days), which allowed the process to run every day.

Also on the SAP ECC side, but in the Eclipse environment, actions had to be taken to define appropriate CDS views that would support the delta mechanism in the context of SAP Datasphere:

• Creating views that return the number of the last request from the PSA table for a given source so that the data exported to SAP Datasphere is limited only to data loaded into the PSA table from the current/last day;
• Creating views containing all fields of the PSA table for a given source for the last request. The views defined in this way were the basis for replicating the necessary data in SAP Datasphere, based on SAPI extractors and taking into account the delta mechanism defined for them.

External SQL databases can also be connected to SAP Datasphere. This can be done with most of the commonly used databases. However, you should check on the SAP website whether the version of your SQL database is supported. Some databases, such as Microsoft SQL Server and Oracle, have built-in connectors already in SAP Datasphere. This means that it is possible to connect them directly. In other cases, the connection of the databases is done via the universal JDBC connector.

The data sources for external SQL databases are the tables defined on them. They can be accessed remotely as well as by loading data depending on the database used and its version. In addition, in each case, the installation of adapters suitable for the existing SQL databases is required on the Cloud Connector side.

Data sources for SAP Datasphere can be cloud solutions not only from SAP, but also from third parties. Examples include Google BigQuery and Google Cloud Storage. Especially the first of these tools seems interesting from the point of view of SAP Datasphere, as it does not require data replication. Data can be accessed from SAP Datasphere completely remotely. What does this mean in practice?

If we have defined Google BigQuery, which collects data from mechanisms such as Google Trends or Google Advertisements, we can use this data directly in SAP Datasphere It is possible to integrate or collate it in combination with data from other sources without having to store it in our data warehouse.

Only selected SAP Datasphere source systems and the method of their integration are discussed above. The list of connectors enabling connection to individual databases or systems is long. Even if a dedicated connector for a particular source system/database cannot be used, it is possible to use a more general mechanism, such as the already mentioned JDBC or Odata service.

In addition, SAP Datasphere has built-in mechanisms for exporting data externally, including to third-party solutions.

All of this makes SAP Datasphere seem an attractive solution for both existing SAP users and those who are just starting their adventure with SAP already in S/4HANA version or who do not have any SAP system at all. There are many possibilities for integration and access to data. It is worth noting that most data can be accessed from SAP Datasphere in remote mode, without the need to load data into the warehouse. The environment interface itself is also user-friendly and graphical, enabling advanced key users to model data.

The term “advanced" means here primarily that they must know where the data they are interested in is stored in the source system, i.e. in which tables or CDS views.