(Glue-2108) Data replication with delta

There is an option to extract data from the source to target using delta load. Delta specifies how data is extracted from source to target. Currently, the delta functionality is supported only for the extraction from the SAP system to storage. For more information about delta see Delta Mechanisms.

To successfully execute delta loads we need to create a SAP Table Fetcher. SAP Table Fetcher provides multiple delta types: TRIGGER, DATE, TMSTMP, VALUE, VALUE_DIST. More information can be found in the chapter Manage Delta.

If we are executing the extraction process using delta, there are three load types we can choose from.

• Delta Init Without Data

• Delta Init + Full Load

• Delta Load

Open

Delta Init Without Data

This load type initializes the delta pointer and performs no load. This is useful for scenarios, where we don't want to transport all data or the data were already transported earlier and now we just want to initialize delta.

For example, if we have a source table with 4 records that we don't want to transport, we will initialize delta without data, see pictures below.

Open

Based on the logs we can see that 0 records were imported into the target table regardless that some data are present in the source table. Now we will add 5 new records to the source table and execute Delta Load.

Based on the logs we can see that only the 5 new records were imported into the target table. And if we look into the target table we can see that only records with ids 5,6,7,8,9 were imported. Delta metadata has also been updated so that we don't import the same records again.

Delta Init + Full Load

This load type performs full load and initializes delta pointer. All data is transported from source to target and delta is initialized. This scenario is useful if we want to initialize delta but also transport all data currently present in the source table.

For example, if we have a source table with 3 records and we execute Delta Init + Full Load all 3 records will be transported to the target, and the delta will be initialized, see pictures below.

Based on the logs we can see that 3 records were imported into the target table. Now we add 5 new records into the source table and execute Delta Load.

Based on the logs we can see that only 5 new records, which we added were transported to the target. If we look into the target table we can see that all the records are present. Delta was also updated so we don't import the same records again.

Delta Load

Performs delta load of all data changes after delta pointer. Delta is then updated. This load type can not be executed without initializing the delta pointer first via Delta Init Without Data or Delta Init + Full Load.