(GLUE-1902) Structured Data Extraction

Owned by SNP
Last updated: Feb 28, 2019 by Martina Pospisilova (Unlicensed)

Structured extraction can only occur when the data structure is formally defined.  An organized data structure helps other technologies such as databases or our programs to easily understand the data.

In this section


Sources:

  • Primary database table
  • External table on a storage (defined by a Glue table in Glue DDIC)
  • BW InfoProviders
  • ABAP code implementing a special interface

Targets:

  • Primary database table
  • An external table in a storage
  • An external folder in a storage (path for a file)

Structured data extractors

Between SAP and external storage, the data movement process is based on extractors which work similar to DTP. The extractors provide various types of data movement:

BW specific extractors

Datavard Glue supports also data replication from BW objects to external storage. For this topic, please refer to the specific SAP BW to storage documentation.



Extractor Insights

In the section below, you can see a more detailed explanation of how to create extractor, and how does the data extraction process work.






Extractor creation

When you're creating an extractor, you must define what data you want to extract, how you want to change it and where you want to save it. 

The whole procedure is described in more detail below.

Data selection

Data selection defines which data will be used for data extraction.

Firstly, you define the origin of data as a technical source. The source can be represented by a DB table or a BW object.

You choose a delta mechanism (so-called delta type) to specify which data is to be extracted. It may be either full load - full content of the data source, or only the data that has changed since the last extraction.

And finally, you define data filter and data package size.

Data transformation

In case the selected data doesn't have the required form for the transfer, it may be transformed.

You can define which source structure fields will be transformed to target structure fields (see Field Mapping for more info).

Also, you can use transfer rules to adjust the data content. The purpose of transfer rules is to change data so that it corresponds to the business requirements (extraction scenario).

Data Transfer

Lastly, you define the target of data extraction.

Data extraction process

The data extraction process is responsible for the extraction of the data.

It is a cycle that consists of multiple phases and ends when all specified data is extracted.

Phase 1: Data selection

This phase loads the data from the source structure based on defined delta and filter. The volume of data is limited by the package size.

If there is no available data, the process terminates.

Phase 2: Data transformation

Based on field mapping, the data is moved from the source structure to the target structure.

After the movement, the data can be changed by Glue transfer rules. These rules are simple ABAP programs. We deliver a default set of transformations (Currency transformation, Meaningful values ...), but you can create your own custom implementation, too. 

Phase 3: Data Transfer

The selected and transformed data is transferred to external storage.

Based on how the target is defined, the data is inserted directly to external storage table or file system.

Phase 4: Go to phase 1