(Glue-2108) Extraction Process

How to Create an Extraction Process Object

1. Run the transaction /DVD/GL80 to open Object Navigator

2. In the context menu of the desired package, select the option Create Extractor 2.0 object
   

3. Enter the name of the new Extraction Process under Object name and a short description about it under Description

4. Select Extraction Process under the Component type dropdown and under Component implementation leave the default implementation

5. Confirm the popup
   

Activation of an Extraction Process

After the creation, the object is in the saved state.

Mandatory parameters to be filled before the activation

Open

• Fetcher

  • name of the Fetcher

  • provided with F4 help

• Consumer

  • name of the Consumer

  • provided with F4 help

Mandatory buttons to click before the activation

• Suggest Transformation Mapping 

  • button in the toolbar of Transformation part 

  • generate mapping based on the filled Fetcher and Consumer

Optional buttons to click before the activation

• Apply currency conversion 

  Open

  

  • button in the toolbar of Transformation part

  • apply currency conversion rule for all or selected fields

• Apply conversion routine 

  • button in the toolbar of Transformation part

  • apply currency conversion rule for all or selected fields

Optional inputs before the activation

• Start Routine

  • button in Transformation toolbar for ABAP code editor

  • executed before the transformation part of the extraction

• End Routine

  • button in Transformation toolbar for ABAP code editor

  • executed after the transformation part of the extraction

• Rule

  • icon in Transformation

  • for more info see the chapter Transfer Rules

Additional Functionalities of Extraction Process

The Extraction Process provides a few other functionalities on top of the Common functionalities of Extractor 2.0 objects.

• Execute extraction

  • button in Object Navigator toolbar

  • execute the report generated by the activation of the Extraction Process

  • for more information see Execute extraction section below

• Monitor

  • button in Object Navigator toolbar

  • option in the floating menu of Extraction Process

  • display detailed information for each executed extraction of the current Extraction Process

  • for more information see Extraction Request Monitor

Execute Extraction

Execution of the extraction is done by the pre-generated report, based on the Execution Process. Therefore each report can contain different input parameters.

The pre-generated report has an execution button and can have 5 sections of inputs:

• Execution type - selection of execution type

  • Standard Execution

  • Mass Execution 

• Basic parameters - section for general settings of the extraction

  • Preview mode

    • when enabled, data will be not written to the target but only the sample will be displayed on the screen

  • Package size

    • number of rows selected for each package of package based source

  • Package size MB

    • input for several MB, with compute button Calculate

• Selection filter - the section is generated based on the selection in the specific Fetcher

• Additional Fetcher config - section generated for editable parameters of the Fetcher

  • Load type

    • a dropdown of all available Load types

  • Use cursor

    • when enabled, data from the source will be processed by the packages

• Additional Consumer config - section generated for editable parameters of the Consumer

  • Data validation (only for Glue consumers)

    • when enabled, the data extracted to the storage will be validated with the source data. To make the checkbox visible in the extraction process, turn on the parameter ENABLE_DATA_VALIDATION in Glue settings (transaction /DVD/GL_SETT).

  • Enable merge

    • the flag that will enable the merge operation

    • can be also enabled/disabled in the Extraction process selection screen

  • Truncate target

    • when enabled, all data will be deleted from the target before writing from the source

  • Preserve staging files in case of failure (only for Glue consumers with Glue table on Hive, Redshift, BigQuery, or Snowflake)

    • the checkbox is enabled by default

    • when enabled, CSV files on the staging location won't be deleted in case the extraction fails on the COMMIT phase - enables investigation and Retry commit functionality

    • for information on how to retry commit using monitor please check Retry commit

By clicking on the execute button a pop-up will appear with a cancel button and two buttons for execution type:

• Dialog - the extraction will be executed in a dialog process

• Background - another pop-up with a schedule will be displayed and after the confirmation, the extraction will be executed in the background process

If the user chooses to schedule the extraction to a particular date and time, a new request is created right away and displayed with a Pending (icon ) status.