(Glue-2111) Create a Glue table on File storage

The creation of the Glue table requires a few additional steps in comparison with creating a table on transparent storage. You can find the common procedure in the chapter Create a Glue Table and during the selection of the Storage ID you need to follow these steps:

1. Fill in a Storage ID with a valid File storage ID. You should see the File storage options button. You might need to press enter (after filling the storage ID) in order to see this button.

2. Open

   

   Press the File storage options button. You should see the following popup

   Open

   

3. This is the default state and if confirmed the logic of the Profile mapping will be used.
   If the second option from the radio buttons Specific profile is picked, the popup will change accordingly:

4. There is a need to use an existing Profile name that will be used, for that the F4 help can be used.
   When the last, third option from the radio buttons Custom options is picked, the popup will change accordingly:

   Only in this case, the next (third) step is necessary.

5. Fill in the parameters based on your requirements or leave the default one, if suitable.

   1. Container name – Directory where files generated by Glue are stored (obligatory parameter). 

      1. You can also specify a path to the container by using a slash sign as a separator between the containers.

      2. There’s also the option of using Glue placeholders in the container name.

   2. File type – File format to store the data (CSV or PARQUET)

   3. CSV options:

      1. Delimiter type – Character which separates values in generated .csv files. This character can be selected from F4 help (obligatory parameter).

      2. Put values into quotes – Option for putting values into the quotes (use when there is a delimiter character present in data).

      3. Use extending escaping (optional) – Replaces escape characters, such as newline, backspace, tabulator, etc., by the 'space' character. This can be customized in the table /DVD/RL_EXT_ESC and a default set of values can be filled by transaction /DVD/RL_DEFAULT_ESC.

      4. Include header (optional) – Includes a header line at the beginning of the file with data.

      5. One file extraction (optional) – Keeps all data in one file (data from one extraction is stored in only one file).

      6. Compress file(optional) – Compresses the file into a gzip. This feature cannot be selected if the parameter One file extraction is enabled and vice versa.

      7. Compression level – Compression level of the gzip file. Available values can be selected from F4 help. The default value is 5. The parameter will appear only if the Compress file option is checked.

   4. PARQUET options

      1. Create Delta Lake metadata - metadata JSON files are created in the “_delta_log” subfolder to allow Delta Lake Spark processing

6. Confirm your selection. Now you can continue with Step 2 of Create a Glue table procedure.

Glue table on File storage explained

On file-based storages, no structures like database tables are created and data is stored within a specific location in .csv files. To keep track of Glue tables created, the .json metadata file is generated at a specific location (defined in Container name) during Glue table activation. A similar metadata file is also transferred in case an alter operation is performed on the Glue table. The naming convention for the Glue table name can be customized via Glue Settings - Binary Storage settings parameter called Binary table metadata file name.

JSON metadata structure

The metadata for each Glue table created on file storage is stored in the JSON file and contains the technical description of the Glue table. This file can be used by an external processing application to process the Glue table correctly.
In the next section, we discuss the JSON metadata structure and the meaning of particular attributes.