Chapter (Glue 2402) Replication to Files explained which files are written to file-based storage:
...
You can edit the naming convention withSNP Glue™ Profile Settings.
Within this chapter, we’ll explain how to use the provided metadata to recognize delta and do proper merges (for example in spark-based applications like Databricks).
...
I: Delta Init Without Data.
L: Delta Init + Full Load.
D: Delta Load.
F: Full Load (Repair) Without Delta Update.
R: Recovery of Previous Deltas - https://datavard.atlassian.net/l/c/X1UGkpNh ODP Fetchers only.
More information can be found in the chapterhttps://datavard.atlassian.net/l/c/7FUaNembData Replication with Delta.
Request metadata - DeltaType
The property DeltaType tells you which delta mechanism was used for reading the data from the source object. In the table below you can see which DeltaTypes you can expect for the used SAP source object:
Objects | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
SAP Object | SAP Table | Table View / Hana View | CDS View | Extractor | BW Object | DSO | aDSO | BEx Query | |||||||
SNP Glue™ Fetcher | SAP Table Fetcher | SAP Table Fetcher Hana View Fetcher | SAP Table Fetcher ODP Fetcher | ODP Fetcher | Listcube Fetcher ODP Fetcher | ODP Fetcher DSO Fetcher | ODP Fetcher aDSO Fetcher | BEx Query Fetcher | |||||||
DeltaType | Indication field (type) | Load type | Insert | Update | Delete | Deduplication | Comment | ||||||||
FULL | X | X | X | X | X | X | X | X | F | No | Full load - no delta | ||||
TRIGGER | X | /DVD/GL_DELFLAG | D, F, I, L | ““ | ““ | D | Yes | Delta is captured by database triggers and stored in a shadow table (keys only). During a delta replication, the shadow table will be deduplicated (latest entries) and joined to the source table to get the data itself. | |||||||
Data Replication with Delta Data Replication with Delta Type Trigger | |||||||||||||||
VALUE | X | X | X | X | D, F, I, L | No | Delta is captured by field values like a creation or change date or an increasing key number - |
Delta Mechanisms. So based on the field you choose you can capture creates and/or changes. Deletes can only be recognized if the table itself has a delete column. Hard deletes on the database level can’t be captured by this method. | |||||||||||||||
VALUE_DIST | X | X | X | D, F, I, L | |||||||||||
DATE | X | X | X | X | D, F, I, L | ||||||||||
TMSTMP | X | X | X | X | D, F, I, L | ||||||||||
CHANGELOG | X | RODMUPDMOD | D, F, I, L | N | ““ | D | Optional - DSO deduplication | Recordmodes can be selected in the |
Advanced DSO Fetcher. Refer to the table below. | |||||||||||||||
REQUEST | X | RODMUPDMOD | D, F, I, L | N | ““ | D | |||||||||
SLT | /DVD/GL_DELFLAG | I | U | D | Optional | A = Archive | |||||||||
ODP_DELTA | X | X | X | X | X | ODQ_CHANGEMODE | D, F, I, L, R | C | U | D | ODQ_CHANGEMODE and ODQ_ENTITYCNTR are available. Refer to the table below ODP Fetchers. |
...
ODQ Changemode and BW Recordmode Overview from SAP
...
Merge to http://delta.io tables
SNP provides a python Python notebook that can be used as a template to show how the merge can be done on delta tables with a spark environment like Databricks or Azure Synapse. The reading of metadata and the creation of merge statements is done dynamically. It will be used like this:
...