(Glue-2108) Release Notes 2108

New Features

HANA Native extraction

HANA Native extraction allows replicating data directly from HANA native objects.

The following objects are supported: SAP HANA Calculation View, Attribute View, and Analytic View.

You can replicate data also in mass execution mode and split the data based on a number of records or on size in MBs.

Extraction Scheduler

Extraction scheduler lets you effectively schedule and execute multiple extractions. Using this tool, it is no longer needed to execute the extraction jobs individually one by one, but you can define a set of extractions and trigger the execution with a single click.

Parquet file support

When using file storage as a target of your replication, you can now choose between CSV and Parquet file format.

Parquet is an open-source file format designed to support fast data processing for complex data. It is a columnar, open-source, and self-describing format that offers several advantages when comparing with CSV, such as file compression, better performance of data queries, or support of schema evolution.

Synapse

You can now connect to Azure Synapse dedicated SQL pool as a target in storage management 

Improvements

Transport Management

Glue Transport Management has been reworked to improve the stability and consistency of Glue transports.
Namely, the following has been changed:

• Versioning of the collected objects

• An advanced activation screen has been introduced to provide more control over the import of the collected objects

• Logging has been enhanced to provide clear and meaningful information

Extractors

• You can extract data from SAP DSO using the newly created Glue DSO Fetcher. This gives you the possibility to use all the features provided by Extractor 2.0 framework.

• Fetcher screens have been adjusted to improve the user experience.
  F1 help information has been added for fetcher parameters, unnecessary parameters have been removed.

• The default package size was changed to 100 MBs to optimize the performance.

• Retry commit functionality is supported for storage that uses the staging location (Apache Hive, AWS Redshift, Google BigQuery, Snowflake).
  When the extraction fails on the commit phase, CSV files transferred during extraction are not deleted. If the error was caused e.g. by connection issues, the commit can be relaunched, saving time that is needed to transfer the files to the staging location.
  The usability of this feature has been significantly improved.

CDC

• Trigger-based delta capture has been optimized to improve the performance on HANA.

• New Glue settings have been introduced to enable smooth extraction in streaming mode.

Glue cleanup

• You can clean old data from Glue queues based on the defined retention settings.

Additional features

• We have created a new tool to troubleshoot problems in case activation or deactivation of Glue tables fails.

• The backup tool enables you to backup the relevant table content before a system refresh has been updated and improved.

Storage Management

• HANA Native connector supports now bulk insert of data. This functionality has improved the performance of data injection by 40 times.

• You can now connect to AWS S3 using S3 managed encryption keys.

Fixes

Extractor

• It was possible to execute mass execution without specifying the partitioning field. This has been corrected and now you can execute it only after all the parameters are properly defined.

• The replication of data from the SAP BEx query was transferring data without leading zeros. The data is now transferred in the correct format.

• When extracting the data from ADSO, the initialization of delta information was not correct and could potentially cause data inconsistency.

• It was not possible to change ListCube Fetcher runtime parameters “Use DB aggregation“ and “Use Cube aggregates“ in the extraction process.

• The execution of extraction in the dialog was failing without a proper message. The error information has been improved the explain clearly the reason for the failure.

CDC

• In case an SAP table had columns that were used in the internal Glue shadow table (TIMESTAMPL, SEQUENCE, TYPE), the extraction failed. This has been corrected and the extraction now works also in such cases,

Object Navigator

When performing actions in the Object Navigator, the tree was refreshed too many times. This has been corrected.

Cleanup Watchdog

Glue Cleanup Watchdog was not working when too many requests were selected for deletion. The logic has been corrected to properly handle also large amounts of requests.