(Glue-2311) Addressing Duplicates during Extraction from SNP Glue™ Queue

Owned by SNP
Nov 24, 2023
1 min read

In case of heavy lifting extraction from SNP Glue™ Queue cluster to storage using Merge functionality, the issue with duplicates might occur for some storages like Azure SQL DB.

Since SNP Glue™ Queue might contain duplicate records, e.g. when the data has been written using SLT Replication Server, there is an option to eliminate those duplicates during extraction from Queue. For that, you need to enable the parameter Key deduplication in Queue Fetcher and after every package is read from Queue, the data will be compared for key uniqueness. Only the latest (actual) values will be left in the processing package, which will subsequently be transferred to the storage.

Using this feature requires a few adjustments of the second part of the Heavy lifting scenario, i.e. Transfer data from SNP Glue™ Queue cluster to Storage(s):

  1. Create Queue Fetcher with the parameter Key deduplication checked.

  2. Create SNP Glue™ Table with additional field GLCHANGETIME.

  3. Create SNP Glue™ Table Consumer with the parameter Enable merge checked.

  4. Create Extraction Process.

  5. Execute extraction.