(Glue-2411) "Heavy lifting" Extraction using Queue

Owned by SNP
Nov 20, 2024

A Heavy lifting scenario can be useful when SAP Table to Storage scenario is not applicable due to constantly running processes on the SAP table, which makes replication of huge amounts of data to external storage time-consuming. This may be e.g. extraction of large volumes of data from the SAP HANA system or from Advanced DSO. During the replication process, the source table is locked and all the operations are blocked. In case such a table is critical for other processes within the SAP system or the customer does not want to postpone BW reporting, the Heavy lifting extraction can be used.

In this section:

This scenario incorporates additional steps into SAP Table to Storage scenario, which saves the data from the SAP table first to a temporary place (Queue cluster) at a minimal runtime and enables the extraction process to perform further transfer to multiple external storages regardless of other processes on the SAP system. 


Below, you can find a step-by-step manual of how to set up and execute the extraction of data between SAP Table > Queue cluster > Storage.

  1. Transfer data from SAP Table to SNP Glue Queue cluster
    1. Create SAP Table Fetcher
    2. Create (Glue-2411) Glue Queue
    3. Create Queue Consumer
    4. Create (Glue-2411) Extraction Process
    5. Execute extraction

  2. Transfer data from SNP Glue Queue cluster to Storage(s)
    1. Create Queue Fetcher
    2. Create Glue Table
    3. Create Glue Table Consumer
    4. Create Extraction Process
    5. Execute extraction

This mechanism is also being used in SLT integration. In this scenario consider Addressing duplicates during extraction from SNP Glue Queue.