(Glue-2208) "Heavy lifting" extraction using Queue

Owned by SNP
Last updated: Aug 23, 2022 by Marian Vidra

"Heavy lifting" scenario can be useful when SAP Table to Storage scenario is not applicable due to constantly running processes on the SAP table, which makes replication of huge amounts of data to external storage time-consuming. This may be e.g. extraction of large volumes of data from the SAP HANA system or from Advanced DSO. During the replication process, the source table is locked and all the operations are blocked. In case such a table is critical for other processes within the SAP system or the customer doesn't want to postpone BW reporting, the "Heavy lifting" extraction can be used.

In this section:

This scenario incorporates additional steps into SAP Table to Storage scenario, which saves the data from the SAP table first to a temporary place (Queue cluster) at a minimal runtime and enables the extraction process to perform the further transfer to multiple external storages regardless of other processes on the SAP system. 


Below, you can find a step-by-step manual on how to set up and execute the extraction of data between SAP Table -> Queue cluster -> Storage.

  1. Transfer data from SAP Table to SNP Glue™ Queue cluster
    1. Create SAP Table Fetcher
    2. Create Glue Queue
    3. Create Queue Consumer
    4. Create Extraction Process
    5. Execute extraction

  2. Transfer data from SNP Glue™ Queue cluster to Storage(s)
    1. Create Queue Fetcher
    2. Create Create SNP Glue™ Table
    3. Create Glue Table Consumer
    4. Create Extraction Process
    5. Execute extraction

This mechanism is also being used in SLT integration. In this scenario consider Addressing duplicates during extraction from SNP Glue™ Queue.