(SM-2405) GCS, BigQuery & BigQuery Streaming Storage Setup

This page provides general guidelines on how to set up GCP Storage and Big Query to work with Storage Management.

 

1. Network prerequisites

To enable communication of SAP systems with the Google Cloud Platform, the following hosts/port needs to be reachable from the SAP system:

Port

Address

Port

Address

443

*.googleapis.com

443

accounts.google.com

2. GCP prerequisites

The entry point to GCP is https://console.cloud.google.com/. Log in with the company account.

2.1 Project

Select a project or create a new one

An overview of the project resources is always accessible by clicking on:

 

 

2.2 Service account

The key component for GCP access from the SAP system will be the Service account with the generated key in JSON format. Download the key file for authentication to GCP later in the configuration.

2.3 Storage bucket

The next step is the creation of a Storage bucket. Markdown the bucket name for later configuration.

2.4 BigQuery dataset

Similarly, the BigQuery dataset needs to be created with appropriate permissions (read/write) for the Service account. The dataset name will be used later.
Java connector uses GC API to run any queries on BigQuery.

2.6 Roles

At a minimum, the following roles are required:

  • BigQuery Job User on the Project level

  • BigQuery Data Editor on the dataset level

  • Storage Object User on the bucket level

3. SAP configuration

3.1 JSON key

Step one on the SAP side is to upload the GCP Service account key file (from step 1.2) to the application server. The standard path for sensitive files used by Java connector is /sapmnt/<SID>/global/security/dvd_conn. All information necessary for authentication is contained within. Access to the file should be limited to authorized personnel (through both operating systems and SAP transactions).

3.2 Java connector

Java connector is a critical middle-ware component. Follow the steps in this guide Java Connector Setup to set it up before you continue. 

3.2.1 Java connector GCP libraries

In addition to standard Java connector configuration, GCP libraries must be used as a dependency in Java Connector Setup.

If the library is missing, contact your representative for further instructions on how to load the library bundle into the system.

3.3 Storage Management Setup

The final step in SAP & GCP connectivity is the creation of two storages in transaction /DVD/SM_SETUP. One for the transfer of flat files, and the other one for table management, and SQL queries.

Google Cloud Storage (GCS) - as seen in the screenshot, the required fields are:

Storage ID

Logical name of the storage connection

Storage type

GCS (stands for Google Cloud Storage)

Bucket name

Name of the storage bucket from step 1.3

Path Prefix

Path to landing area within GCS bucket

Configuration path

Physical path to JSON key file from step 2.1

Java connector RFC

TCP/IP RFC destination used for communication with Java connector (configured in step 2.2)

JDBC Call Repeat

Number of times failed calls should be retried

JDBC Repeat delay

The delay between retried calls

Compute hash

Calculate the hash of data before and after the transfer

 


Google BigQuery storage (BIGQUERY) - required fields:

Storage ID

Logical name of the storage connection

Storage type

BIGQUERY

Dataset

Name of the BigQuery dataset from step 1.4

Reference Storage

Google Cloud storage used for the flat-file transfer

Java connector RFC

TCP/IP RFC destination used for communication with Java connector

JDBC Call Repeat

Number of times failed calls should be retried

JDBC Repeat delay

The delay between retried calls

Use extended escaping

Checked if extended escaping should be used (replaces escape characters, such as newline, backspace, tabulator, etc., with the space character)

Force file cursor reader

Option to force usage of files for data transfer when reading data stored in BigQuery

 

 

Google BigQuery Streaming storage (GBQ_STREAM) - required fields:

Storage ID

Logical name of the storage connection

Storage type

GBQ_STREAM

Dataset

Name of the BigQuery dataset from step 1.4

Configuration path

Physical path to JSON key file from step 2.1

Java connector RFC

TCP/IP RFC destination used for communication with Java connector

JDBC Call Repeat

Number of times failed calls should be retried

JDBC Repeat delay

The delay between retried calls

 

 

If all configuration steps are completed and there is no problem with permissions, an automatic storage check shows a green tick.