(Glue-2311) Data Validation

Owned by SNP
Nov 24, 2023
3 min read

SNP Glue provides two mechanisms for the data validation to cover the entire replication scenario from SLT to the target platform.

  1. Verification of whether the data obtained from SLT is correctly stored in internal SNP Glue tables

  2. Validation of whether the data read from Glue queues matches the data in the target platform

SLT Validation

In this step, we compare the data obtained from SLT to the data stored internally in SNP Glue. The goal is to verify whether SNP Glue stores the data correctly in its original format.

Technical details

Every common field of every line is compared. Generic programming is used, meaning this process is very expensive and should not be used for typical scenarios, only for testing the data inconsistency problems.

Configuration

  • Validation check can be enabled in table /DVD/GL_SLT_PAR

  • You can modify it using /DVD/RL_TM transaction

    • add param_name 'VALIDATION:MT_ID;SOURCE_TABLE'.

    • Possible param_value configurations are ';' - every MT_ID,   '<MT_ID>;' - every table in specified MT_ID,   '<MT_ID>;<SOURCE_TABLE>' - only specified table in specified MT_ID

Example of /DVD/GL_SLT_PAR configuration that will run validation only for table 'SFLIGHT' in MT_ID '02D':

In case the validation fails, an application log is created and the extraction fails.

SLT Data Inconsistency check

In case the corrupted data is being replicated, we have introduced the data inconsistency check. This check is helpful for the analysis. Using it, you can debug the code and identify the root cause of the behavior.

Configuration

Configurable only using table content.

  • FIELDNAME supports asterisk [*]

  • VALUE - this or a higher value is considered inconsistent

  • TIMEOUT is in minutes, and can also be 0 to fail the extraction immediately

Only 'P' (packed number/decimal) fields are checked.

When the inconsistency check is triggered, a log is created under external ID MT_ID->TABLE_NAME INCONSISTENCY

In case the timeout is reached, the extraction fails.

 

Target data Validation

This data validation controls whether the data read from Glue queues matches the data in the target platform.

Technical details

The data copied to the storage is verified with the source data using the XOR hash function. Based on the type of target storage the data is either verified by packages after every write operation or at the end of the extraction.

Please be aware that this logic slows down the overall performance of the data extraction.

Configuration

The validation can be enabled manually for specified extraction processes.

To do so, use the following step list.

  1. Enable the Data Validation option in Glue Settings by setting the ENABLE_DATA_VALIDATION parameter to value 'X'.

  2. In the extraction process, click Execute

  3. Enable Data Validation and save the extraction process variant

  4. In the Glue Scheduler, choose the created variant.

Results

In case the validation passes, the extraction is successfully finished.

In case the validation fails, the extraction ends with an error.