(SM-2508) Setting up Google Cloud Storage and Google BigQuery
This page provides general guidelines on setting up GCP Storage and BigQuery to work with Kyano Storage Management.
Network prerequisites
Step 1: Set up the port and hosts
To enable the communication of SAP systems with Google Cloud Platform, the following ports and hosts (addresses) need to be reachable from the SAP system:
Port | Address |
|---|---|
443 | *.googleapis.com |
443 | accounts.google.com |
GCP prerequisites
Step 2: Log in to GCP
To start working with GCP, visit the GCP homepage and log in using your company account.
Step 3: Select or create a project
Select an existing project or create a new one.
You can gain an overview of the project resources by clicking Google Cloud Platform in the top left-hand corner.
Step 4: Create the service account key
The key component for GCP access from the SAP system will be the service account with the generated key in JSON format. Download the key file for authentication to GCP later in the configuration.
Step 5: Create a storage bucket
The next step is to create a storage bucket. At a minimum, you must assign the Storage Object Admin role to the service account. Note down the bucket name for later configuration.
Step 6: Create a BigQuery dataset
Create a BigQuery dataset for the service account with the appropriate read and write authorizations. The dataset name will be used later.
The SNP Kyano Java Connector utilizes the Google Cloud API to execute queries on BigQuery, which require at least the roles BigQuery Job User and BigQuery Data Owner to be assigned to the service account.
Step 7: Roles
At least the following roles are required for the service account:
BigQuery Job User at the project level
BigQuery Data Editor at the dataset level
Storage Object User at the bucket level
SAP configuration
Step 8: Upload the JSON key
On the SAP side, upload the GCP service account key file from step 4 to the application server. The standard path for sensitive files used by the SNP Kyano Java Connector is /sapmnt/<SID>/global/security/dvd_conn. All information necessary for authentication is contained within. Access to the file should be limited to authorized personnel (through both operating systems and SAP transactions).
Step 9: Configure the SNP Kyano Java Connector
The SNP Kyano Java Connector is a critical middleware component. Follow the steps in the chapter (SM-2508) SNP Kyano Java Connector Setup before you continue.
Step 10: Set up Kyano Storage Management
The final step in connecting SAP and GCP is to create two storages in transaction /DVD/SM_SETUP. One storage is for transferring flat files, while the other storage is for table management and SQL queries.
Google Cloud Storage (GCS) - required fields:
Storage ID | Logical name of the storage connection |
Storage type | GCS (Google Cloud Storage) |
Bucket name | Name of the storage bucket from step 5 |
Path Prefix | Path to the landing area within the GCS bucket (e.g., prefix/). Note: The parameter must not start with |
Configuration path | Physical path to JSON key file from step 8 |
Java connector RFC | TCP/IP RFC destination used for communication with the SNP Kyano Java Connector (configured in step 9) |
JDBC Call Repeat | Number of times failed calls are retried |
JDBC Repeat delay | Delay in seconds between retried calls |
Compute hash | Option to calculate the hash of the data before and after the transfer |
Google BigQuery storage (BIGQUERY) - required fields:
Storage ID | Logical name of the storage connection |
Storage type | BIGQUERY |
Dataset | Name of the BigQuery dataset from step 6 |
Reference Storage | Google Cloud Storage used for the flat-file transfer |
Java connector RFC | TCP/IP RFC destination used for communication with the Java connector |
JDBC Call Repeat | Number of times failed calls are retried |
JDBC Repeat delay | Delay between retried calls |
Use extended escaping | If selected, extended escaping is used. This replaces escape characters, such as newline, backspace, tabulator, etc., with the space character. |
Force file cursor reader | If selected, data from BigQuery is unloaded into Google Cloud Storage in the form of files and then read from there |
Wrap values in staged CSV files | Wraps values in CSV field in double quotes |
Google BigQuery Streaming storage (GBQ_STREAM) - required fields:
Storage ID | Logical name of the storage connection |
Storage type | GBQ_STREAM |
Dataset | Name of the BigQuery dataset from step 6 |
Configuration path | Physical path to JSON key file from step 8 |
Java connector RFC | TCP/IP RFC destination used for communication with the SNP Kyano Java Connector (configured in step 9) |
JDBC Call Repeat | Number of times failed calls are retried |
JDBC Repeat delay | Delay between retried calls |
Use CDC | Enable/disable CDC and its automatic processing. More information can be found herehttps://cloud.google.com/bigquery/docs/change-data-capture. If it is enabled, GLDELFLAG or equivalent field must be present in the Glue table. |
Max staleness | Relevant only when Use CDC is selected. It defines the maximum staleness of data on BigQuery; more information can be found herehttps://cloud.google.com/bigquery/docs/change-data-capture#manage_table_staleness. |
Once all the configuration steps are complete and if the authorizations are correct, an automatic storage check is performed. If it is successful, a green tick is displayed.