S3 and Redshift storages can both be used as a storage layer for SNP Glue™ or SNP OutBoard™ ERP Archiving. S3 provides inexpensive storage for landing zone or cold archive storage purposes. Redshift comes at a higher cost but can be used as a directly query-able analytical storage, or a hot archive suitable for more aggressive SAP archiving strategies. 

It is recommended to deploy AWS resources in a way, that the development, test, and production environments are isolated.  This means that when you archive or replicate data to S3, there should be 1 S3 bucket per SAP system. With Redshift deployment, a typical scenario would be 1 cluster with separate databases for development and quality environments, and 1 cluster for the production SAP environment.

While the setup itself is simple, from experience we see that the setup usually takes around 2 weeks since multiple teams need to be involved to fulfill the prerequisites (SAP Basis, Network, AWS).

The person responsible for the setup should have general knowledge of AWS S3 and AWS Redshift, SAP basis, and networking basics. You will need to deploy resources on AWS and create access credentials, import our transports to the SAP system, install JRE and JDBC drivers on SAP application servers, and make sure that SAP and AWS environments can communicate. 

General Prerequisites

SAP NetWeaver release

Storage management requires SAP NW 7.01 SP15 or higher.

Open Ports

In a controlled network environment, it is common to have firewall rules in place. To enable communication between SAP systems and AWS, an outbound communication from the SAP system to following ports on the AWS side need to be allowed:

Example of a simple telnet connectivity test:

sapserver01:/ # telnet s3.eu-central-1.amazonaws.com 443
Trying 3.5.139.101...
Connected to s3.eu-central-1.amazonaws.com.
Escape character is '^]'.
^]
telnet> q
Connection closed.

Storage Management allows encrypted communication through the public internet with S3 and Redshift, but for production deployment, it is recommended to have some kind of secure connectivity in place (VPN).

Please refer to AWS documentation for more details.

Java connector

Java connector is a critical middle-ware component used for both S3 and Redshift storage. Please follow the steps from the chapter (SM-2211) Java Connector Setup to set it up before you continue.

Make sure that your Java connector includes libraries with AWS SDK.

S3

This chapter describes the steps to establish a connection to S3 storage.

S3 storage is usually used in the following scenarios:

• Landing zone for raw SAP data from SNP Glue™
• Archiving storage for SNP OutBoard™ Data Tiering (SARA archives or tabular data when used as transparent binary storage)
• Intermediate storage used in combination with AWS Redshift

S3 prerequisites

S3 bucket

You must identify a suitable S3 bucket on AWS or manually create a new one in your AWS subscription based on your requirements. We recommend keeping public access disabled and enabling server-side encryption for added security. Storage management supports both SSE-S3 and SSE-KMS server-side data encryption options.

Note down your S3 region, bucket name and KMS key ID (optional) as this information will be required during storage setup.

If you use Customer Managed Key, make sure to update Key Policy to allow key use for the user/role (Principal) accessing the S3 bucket.

AWS User for programmatic access

We recommend creating a distinct user for every SAP system connected to the AWS services in order to isolate each system's data. Please refer to AWS documentation for best security practices. 

A credentials pair needs to be generated ("access_key_id", "secret_key_id"). Make sure to rotate access keys regularly.

These credentials will be used in Storage Management for read/write access to the S3 bucket. If Redshift is used, it will also be used to load/unload data from the Redshift cluster.

S3 Policy

Assign an appropriate policy to the technical user, so it can access the S3 bucket. Please follow the policy of least privilege. 

For a standard bucket without custom encryption keys, the following policy is sufficient.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::MY-ACCOUNT-ID:user/MY-USER"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::MY-BUCKET/*",
                "arn:aws:s3:::MY-BUCKET"
            ]
        }
    ]
}

S3 storage in Storage Management

Storage Management facilitates transparent communication with different types of storages.

Create S3 storage through the transaction:

/DVD/SM_SETUP

Entries explained:

• Storage ID: Name of the storage
• Storage Type: Choose AWS_S3 for S3
• Description: Extended description of the storage for easier identification
• AWS Bucket name: Name of the existing bucket in S3
• AWS Region: Region where the bucket exists (the recommendation is that also Redshift cluster exists in the same region)
• Path Prefix: Path to the landing area within the AWS bucket
• Custom endpoint: Optional parameter to specify the S3 VPC endpoint URL, example: https://sample-bucket.bucket.vpce-0a2509460a648e95d-x8m26cz5.s3.eu-central-1.vpce.amazonaws.com
• AWS Credentials = None (option to choose for having Access key or None credential type): Special case, when authentication against AWS services is performed by default credentials provider chain. So basically when the user set this option to None, the Java Connector and its Amazon SDK are searching for AWS credentials on the hosting server. In the case of the scenario Assume Role = Assume Role, these credentials are needed to authenticate against AWS Security Token Service.
  
• AWS Credentials = Access key (option to choose for having Access key or None credential type)
  • AWS Access Key: Security information "access_key_id" - this option is required when AWS Credentials is set to Access key
    
  • AWS Secret Key: Security information "secret_key_id" - this option is required when AWS Credentials is set to Access key
    
• Assume Role = None (option to choose from having Assume Role or None assume role type): Not using any "external AWS role" when accessing Amazon resources, the authenticated user uses its policies and attached permissions when accessing Amazon resource
  
• Assume Role = Assume Role (option to choose from having Assume Role or None assume role type):
  • Special case when an authenticated user tries to "lease"/assume an Amazon role and gets that role temporary credentials and user temporary credentials when accessing Amazon resources. The attached permissions to the assumed role are used to evaluate access right on Amazon resources instead of users' permissions. More info can be found here.
    
  • RoleARN: Name of the assumed role in ARN format (example: arn:aws:iam::683735966288:role/dvd_s3_read_role_to_be_assumed) - this option is required when Assume Role is not set to None
  • ExternalID: External ID used to identify against the assumed role, the user has to set proper external ID to be able to successfully get temporary credentials from STS AssumeRole API, more info - this option is required when Assume Role is not set to None
  • RoleSessionName: Identifier for the assumed role session, the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons, can be used for logging on Amazon - this option is required when Assume Role is not set to None
  • AWS STS endpoint: Optional – URL of AWS STS endpoint, when specified regional AWS STS endpoint is used instead of the global endpoint - more info
  • AWS STS region: Mandatory if the AWS STS endpoint is specified, Region related to the specified endpoint 
• Java connector RFC: TCP/IP RFC destination for communication with Java connector
  
• Java call Repeat: Number of times failed calls should be retried
• Repeat delay: Delay between retried calls

• AWS KMS KeyID: Key ID of the key used to encrypt data on S3 (optional)

• Compute hash: Calculate the hash of data

  Path for TMP files must be visible for the instance of the java connector. In case your SAP system is a cluster consisting of multiple physical machines, you need to configure NFS (Network File System). Performing this step you'll make sure that all application servers will be writing temporary data into one shared location, which is visible for the Java connector instance. With this configuration, you will be able to perform storage operations on S3 storage regardless of the actual SAP application server. /sapmnt is usually the NFS directory shared among all SAP application servers.

Complete the creation of the storage by confirming (F8).

Redshift

AWS Redshift is ideal as an archive for frequently accessed tabular data for SNP OutBoard™ Data Tiering or as analytics storage for SNP Glue™.

Redshift prerequisites

S3 storage

Redshift storage requires existing S3 storage to work. Make sure that you finished the steps in the S3 section and that you have working storage ready. 

Cluster and database

You must create a Redshift cluster, which will host your Redshift database. The sizing highly depends on the use case and the amount of data that will be stored there, so please use the "Help me choose" option on the cluster creation page to properly size the cluster.

It is highly recommended that it shares a region with your S3 bucket.

Redshift schema and database user

Create a schema and a database user that Storage Management will use.

In this example, the SID of the SAP system is DVQ and contains recommended naming conventions. Adjust the SQL statements to fit your environment. 

--create user and schema
create user datavard_dvq password 'my-difficult-password';
create schema sapdvq;
alter schema sapdvq owner to datavard_dvq; 
-- assign required permissions to system tables
grant select on pg_catalog.SVV_TABLE_INFO to datavard_dvq;
grant select on pg_catalog.PG_TABLE_DEF to datavard_dvq;

JDBC Drivers

JDBC protocol is used to connect to AWS Redshift. AWS Redshift JDBC driver (RedshiftJDBC42-no-awssdk-1.2.16.1027.jar) must be manually stored on the operating system and be accessible to the connector.

It is recommended to use the default path as in the example below to utilize the predefined logical paths in SAP. Make sure that <sid>adm:sapsys is the owner of the directory dvd_conn and all its contents.

The AWS Redshift storage is created in a similar way to the process of setting up the S3 storage.

Entries explained:

• Storage ID: Name of the storage
• Storage Type: Choose REDSHIFT storage type
• Description: Extended description of the storage for easier identification
• Referenced Storage: Defines which S3 storage will be used by Redshift
• Java connector RFC: RFC connection to Java connector
• Redshift host: Redshift server hosting the Redshift service
• Port: Port number on which Redshift service is accessible
• Database name: Name of DB in Redshift cluster
• Database schema: Name of the schema (usually public)
• Enable update: If checked, delta loads will be merged into existing data
• Use extended escaping: If checked, exotic newline characters in data are escaped
• Driver engine: Use Amazon Redshift
• Driver Classname: Classname of the driver used for loading (the recent version is com.amazon.redshift.jdbc41.Driver)
• Driver path: Logical name of the driver directory
• Username: Redshift user created in the Redshift user group
• Password: Password for specified username
• Login timeout (seconds): Threshold for JDBC timeout
• Enable SSL: Checked if SSL authentication should be used
• SSL Mode: There are two options for SSL mode:
  verify-ca (default option, verifies that the certificate comes from a trusted CA)
  verify-full (both CA and hostname listed in the certificate are verified)

Finish the creation of the storage by confirming (F8). If the SAP system is able to authenticate against AWS Redshift and receives the expected result of the SQL command 'use database', the creation of the storage is considered successful.