(GLUE-1905) SAP to Storage

Owned by SNP
Feb 28, 2019
2 min read

During this extraction data is inserted from SAP into a target storage. The storage can be any technology capable of storing data which is supported by Datavard (GLUE-1905) Storage Management.


Extraction process

During data extraction process occur the following important processes:

  1. Data manipulation

    Each extraction process uses data packages. We transfer data in packages of a defined size, in a one by one order to a storage. A transfer of one package to a storage is a cycle of processes. The first process is data manipulation which is composed of two main parts – data selection and data transformation.

    1. Data selection

      First, you define the origin of data as a technical source. The source can be represented by a database table or a BW object. You should also choose a delta type to apply business-relevant requirements for this selection.
      And finally, you also define the so-called 'Variant' which specifies which data is transferred and the package size.

    2. Data transformation

      Source data doesn't always have the required form for the transfer. The user can use our Rules to specify the final data.
      Rules are used for data transformation. Rules do not specify data selection and cannot influence the data amount. Their purpose is to modify data, so it corresponds to the business requirements (extraction scenario).

  2. Storing data process

    After the data is extracted from a source table (including data manipulation, selection, and transformation), it is stored on an external storage.The process may differ depending on the storage type:

    1. Transparent storage

      Data is moved to a table on an external storage.

    2. Binary storage

      Data is stored on this storage in .csv files, where each file represents one data package (defined by the package size). After the transfer of all the data, a .json file containing extraction metadata is generated.
      The structure of the data stored in .csv is described in a separate .json file which is created after the Glue table activation.