Scripts usage

Scripts are usually used for the following purposes:

• ETL (extract, transform, load) for Hadoop data
• Integration of advanced analytics with existing data flow – run advanced analytics on top of data in Hadoop triggered by process chains
• Joining diverse datasets to gain valuable business insight
• Performing real-time, complex queries on datasets

Datavard Glue Script Editor supports an application for these services:

• Pig
• Hive
• Impala

Initial Screen

Create Script

Use the following steps to create a Glue Script:

1. Enter the script name and press Create

If a script with such a name already exists, a warning message is displayed.

2. Fill in Description and Script type

Following types are available - Hive, Impala, and Pig

3. Specify OOZIE RFC destination - connection to Oozie

Oozie is a workflow scheduler system to manage Apache Hadoop jobs. It is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop, and Distcp) as well as system-specific jobs (such as Java programs and shell scripts).

4. Specify HDFS RFC destination - connection to HDFS (Hadoop filesystem)

5. Create a body of the script

6. Save and activate

Display and edit script

From the initial screen, you can display or edit the existing script.

Activated script cannot be edited, you have to deactivate it first to perform the changes.

Execute a script

You can execute a script from the Initial Screen either in dialog or as a background job.

A script can be also executed in the dialog from the Display/Edit screen, just press Execute icon or F8.
When executing the script in a dialog, a popup with log information comes up after the execution is finished.

The job must be specified. The same principles as described in chapter 5.1.3 must be applied. Likewise described in the mentioned chapter, the information about jobs can be seen in the Simple Job Selection screen. The navigation to the screen is triggered by clicking on the "Job Overview" button in the toolbar.
By clicking on "Script Log" button, the SAP Analyse Application Logs transaction is open with predefined parameters so that it searches for Glue script jobs only. Using this functionality it is possible to see in more detail the execution details and investigate potential problems.

When executing the script in the background, you can use the SAP Analyse Application Logs transaction to see the logs. You can access it also using Script Log button.