(Glue-2405) App Settings

Owned by SNP
May 21, 2024
2 min read

Source settings

image-20240411-101356.png

  1. Controller task scheduling period.

    1. This task for example executes delta based merge.

  2. Backup retention period: Retention period for backups created after table deactivation (in days).

  3. Cleanup staging tables: Turn on/off of staging tables.

  4. Stage table data retention period: Retention period for already merged data in staging table.

  5. Backup retention for Statistics: For how long statistics are being kept.

Merge settings

image-20240411-101327.png

  1. Warehouse: What warehouse should be used for merge execution.

  2. Suffixes for merge objects. Resulting name for the merge object is <Staging table name>+<suffix>. It can be defined for following objects:

    1. Merge table

    2. Merge view

    3. Merge stream

    4. Merge task

  3. Use custom schema for merge objects: Generate merge objects in their respective schema instead of the staging table schema. Defined schema cannot exist as a source schema or be defined for another source schema. It can be defined for following objects:

    1. Merge table

    2. Merge view

    3. Merge stream

    4. Merge task

  4. Merge scheduling period: How often merge should run on replicated tables.

  5. Delete type:

    1. Soft - Keep deleted entries in the merge table with delete flag.

    2. Hard - Do not transfer deleted entries into merge table.

  6. Display soft deletes in merge view: Display entries with delete flag in merge view.

  7. Delta-size based merge: Use stream row count as a trigger for merge.

    1. Delta size to trigger merge: Stream row count needed to trigger merge.

Metadata settings

 

  1. Timestamp field: Name of the field used to determine the last record of a specific key.

  2. Change field: Type of change of specific field.

  3. Insert flag: What identifier in metadata should represent inserts, can insert multiple values if they are separated by a comma, use the '<empty>' placeholder for space.

  4. Update flag: What identifier in metadata should represent updates, can insert multiple values if they are separated by a comma, use the '<empty>' placeholder for space.

  5. Delete flag: What identifier in metadata should represent deletes, can insert multiple values if they are separated by a comma.