(Glue-2105) "Heavy lifting" extraction using Queue

Owned by SNP
May 14, 2021

"Heavy lifting" scenario can be useful when SAP Table to Storage scenario is not applicable due to constantly running processes on the SAP table, which makes replication of huge amounts of data to external storage time-consuming. This may be e.g. extraction of large volumes of data from the SAP HANA system or from Advanced DSO. During the replication process, the source table is locked and all the operations are blocked. In case such a table is critical for other processes within the SAP system or the customer doesn't want to post-pone BW reporting, the "Heavy lifting" extraction can be used.

This scenario incorporates additional step into SAP Table to Storage scenario, which saves the data from the SAP table first to a temporary place (Queue cluster) at a minimal runtime and enables the extraction process to perform the further transfer to multiple external storages regardless of other processes on the SAP system. 


Below, you can find a step-by-step manual of how to set up and execute the extraction of data between SAP Table -> Queue cluster -> Storage.

  1. Transfer data from SAP Table to Glue Queue cluster
    1. Create SAP Table Fetcher
    2. Create (Glue-2105) Glue Queue
    3. Create Queue Consumer
    4. Create (Glue-2105) Extraction Process
    5. Execute extraction

  2. Transfer data from Glue Queue cluster to Storage(s)
    1. Create Queue Fetcher
    2. Create Glue Table
    3. Create Glue Table Consumer
    4. Create Extraction Process
    5. Execute extraction