(Glue-2105) Release Notes 2105

New Features

Glue Queue

New Glue object - Glue Queue was introduced that enables two-step data replication of SAP data. The data is in the first step temporarily stored in a queue. In the second step, it is copied into the target data platform(s) and then automatically cleaned up from the queue. Several data replication scenarios can leverage this functionality like copying the data from the queue into multiple target destinations while accessing the SAP data source only once, or enabling automatic parallel data replication while minimizing the time during which the SAP source object is locked.

ListCube extraction

Listcube extraction allows replicating data from BW objects using a similar approach as SAP transaction LISTCUBE. It supports multiple objects that can be specified as a source.

The following objects are supported: Standard DSO, Advanced DSO, InfoCube, Multi provider, Virtual provider, Open ODS view, and HANA Composite provider.

File profiles

File profiles enable defining how the data is replicated into file storages like Azure ADLS2, AWS S3, Google Cloud Storage, and others.

You can specify file name patterns, delimiter, wrapping of values, and other more advanced settings, and apply it globally, for specific storage, or on a Glue table level.

Automatic adjustment of SAP table structures

A new Watchdog was created which periodically checks pairs of DDIC Table and Glue Table for differences in the structure. If found, it automatically alters all the relevant Glue objects.

Improvements

BEx Query Automatic Mass Execution

• BEx Query Fetcher (BICS) enables automatic mass execution. The logic splits the BEx Query data automatically into smaller chunks based on selected query variables and avoiding memory overflow issues.

Extractors

• Extraction Fetcher parameters can be changed without deactivation and therefore without losing delta information.

• Glue offers cross-client data replication, so data can be replicated from all the clients of an SAP table.

• New transfer rule Xfeld to Boolean function was created to translate ABAP XFELD or FLAG fields to BOOLEAN expressions.

• When using the deletion flag to capture entries that have been removed, Snowflake and MS SQL storages can utilize the merge functionality. This merges the data leveraging the deletion flag of the row and deciding if the entries should be deleted or inserted/updated.

• Delta Watchdog was enhanced to support data extraction from Glue Queues. Double-clicking on the Job Name will redirect a user to SM37 with predefined parameters.

Object Navigator

• The distribution of objects in the Object Navigator tree can be changed using Logical Folders. Folders can be created under the package and Glue objects can be assigned to them.

Selective Deletion

• Selective deletion is now available for Snowflake storage.

Storage Management

• New Direct JDBC reader was implemented for Hive, Impala, and Azure Databricks storages. This reader uses JDBC calls instead of temporary files CSV and improves the reading performance by approx. 40%.

Fixes

Glue Cockpit

• Glue cockpit performance has been improved, the calculations of numbers of running jobs have been optimized.

Data Dictionary

• ALTER operation on a Glue table with transparent-binary storage didn’t generate an updated JSON metadata file. This has been corrected.

Data Browser

• Data Browser was not working on file storages. This has been corrected and generated files can be now be seen on the target storage.

• Comparison operators <, <=, >, >=, !=, = and exclude ranges didn’t work for the following storages: BigQuery, Redshift, Snowflake, and SIQ. This has been corrected.

Glue Database Utility

• Glue Database Utility was deleting wrong data during the cleanup procedure. This has been corrected and now only relevant data is deleted.

Extractor 2.0

• Extraction process was failing after the source SAP structure was modified. With the latest release, this has been corrected. In the case of the standard scenario, only the original data structure (as defined in a Glue table) is replicated.

• In some cases, the BEx Query fetcher wasn’t providing any data even though BEx Query contained it. This has been solved by repeating the request to the OLAP engine.

• Extraction process using SAP Table Fetcher and load type “Full Load” or “Delta Init” was updating the delta information even in case of failure when no data has been transferred.

CDC

• It was possible to execute an extraction process in delta mode even if it was already running. This has been disabled.

• When the delta watchdog was stopped, it was displayed as "failed" in the SM37 transaction.

• Trigger delta on views was not working correctly and in some cases, duplicated records were extracted. This has been corrected.

Glue Monitor

• Extraction process was displayed as “running” in the Glue extraction monitor even if it has failed or when executed in a preview mode. This has been corrected.

• Glue monitor filter was not working properly after refresh. This has been corrected.

Transport Management

• The filter definition of Glue fetchers was not transported correctly.