(Glue-2408) Create SNP Glue™ table on File storage

The creation of the SNP Glue™ table requires a few additional steps in comparison with creating a table on transparent storage. You can find the common procedure in the chapter Create SNP Glue™ Table and during the selection of the Storage ID you need to follow these steps:

1. Fill in a Storage ID with a valid File storage ID. You should see the File storage options button. You might need to press enter (after filling in the storage ID) in order to see this button.

2. Open

   

   Press the File storage options button. You should see the following popup.

   Open

   

   1. Container name (Optional): If specified, the Container name will not be generated by the profile. (For more information about the Container name refer to point 5a.)

3. This is the default state and if confirmed the logic of the Profile mapping will be used.
   If the second option from the radio buttons Specific profile is picked, the popup will change accordingly:

   1. Container name (Optional): If specified, the Container name will not be generated by the profile. (For more information about the Container name refer to point 5a.)

4. There is a need to use an existing Profile name that will be used, for that the F4 help can be used.
   When the last, third option from the radio buttons Custom options is picked, the popup will change accordingly:

   Only in this case, the next (third) step is necessary.

5. Fill in the parameters based on your requirements or leave the default one, if suitable.

   1. Container name: Directory where files generated by SNP Glue™ are stored (obligatory parameter). 

      1. You can also specify a path to the container by using a slash sign as a separator between the containers.

      2. There is also the option of using SNP Glue™ placeholders in the container name.

   2. File type: File format to store the data (CSV or PARQUET or AVRO).

   3. CSV options:

      1. Delimiter type: Character which separates values in generated .csv files. This character can be selected from F4 help (obligatory parameter).

      2. Put values into quotes: Option for putting values into the quotes (use when there is a delimiter character present in data).

      3. Use extending escaping (optional): Replaces escape characters, such as newline, backspace, tabulator, etc., with the space character. This can be customized in the table /DVD/RL_EXT_ESC and a default set of values can be filled by transaction /DVD/RL_DEFAULT_ESC.

      4. Include header (optional): Includes a header line at the beginning of the file with data.

      5. One file extraction (optional): Keeps all data in one file (data from one extraction is stored in only one file).

      6. Compress file(optional): Compresses the file into a gzip. This feature cannot be selected if the parameter One file extraction is enabled and vice versa.

      7. Compression level: Compression level of the gzip file. Available values can be selected from F4 help. The default value is 5. The parameter will appear only if the Compress file option is checked.

   4. PARQUET options:

      1. Create Delta Lake metadata: Metadata JSON files are created in the _delta_log subfolder to allow Delta Lake Spark processing.

6. Confirm your selection. Now you can continue with Step 2 of Create SNP Glue™ table procedure.

SNP Glue™ table on File storage explained

On file-based storages, no structures like database tables are created and data is stored within a specific location in .csv files. To keep track of SNP Glue™ tables created, the .json metadata file is generated at a specific location (defined in the Container name) during SNP Glue™ table activation. A similar metadata file is also transferred in case an alter operation is performed on the SNP Glue™ table. The naming convention for the SNP Glue™ table name can be customized via the SNP Glue™ Settings - Binary Storage settings parameter called Binary table metadata file name.

JSON metadata structure

The metadata for each SNP Glue™ table created on file storage is stored in the JSON file and contains the technical description of the SNP Glue™ table. This file can be used by an external processing application to process the SNP Glue™ table correctly.
In the next section, we discuss the JSON metadata structure and the meaning of particular attributes.