...
| Code Block |
|---|
hive.exec.dynamic.partition = true hive.exec.dynamic.partition.mode = nonstrict |
Example:
Hadoop user
We recommend to create distinct users for every SAP system connected to the Hadoop cluster in order to isolate each system's data.
There is usually a central repository for Hadoop users (LDAP/AD), but you can also create the user locally (on every Hadoop cluster node).
...
Example of custom SAP role in PFCG transaction (Display Authorization Data):
SSL for SAP RFCs
...
The HTTP service has to be active in SAP system. It can be checked via transaction
SMICM > [Goto] > Services
There are two particularly important parameters affecting HTTP communication of SAP system:
...
The name and description of the destination is optional, but it is recommended to designate its purpose with keywords 'Hadoop' and 'HttpFS'. In our example, the RFC destination also contains the Hadoop server hosting HttpFS service for the sake of clarity:
Entries explained:
...
If SSL is used: It’s necessary to enable SSL and add client certificate list to be used in Logon & Security tab.
Authentication RFC
...
The RFC setup is very basic as parameters affecting authentication are defined elsewhere. It is recommended to use the generic RFC name 'HADOOP_AUTH_CONN':
Entries explained:
...
Java RFC by name refers to the Java service which is used for communication with Hadoop services. Again, the setup is basic, and parameters of the Java connector are defined in separate tables:
Entries explained:
...
The first step is to map logical path ZHADOOP_SECURITY to the OS path where the files are stored. The actual OS path is created in the section Datavard connector directories (/sapmnt/<SID>/global/security/dvd_conn).
Kerberos logical file definition
...
When the logical path is defined, file definition follows:
ZHADOOP_KRB_KEYTAB and ZHADOOP_KRB_CONFIG refer to Kerberos keytab of <sid>hdp user and Kerberos configuration file defined in section Kerberos keytab and configuration files respectively. ZHADOOP_CDH_DRIVER refers to the custom Cloudera driver configuration file, which will be generated during the storage activation.
...
If Hadoop services cluster resides in safe environment which is accessible only with SSL authentication, the following logical file needs to be defined as follows:
Drivers logical file definition
...
In our example, we will be using Hive and Impala JDBC Drivers provided by Cloudera. The first step is to map logical path ZJDBC_DRIVER_PATH to the OS path where the files are stored (in our case /urs/sap/<SID>/dvd_conn/drivers/).
Example:
When the logical path is defined, a definition of driver specific folders follows:
ZJDBC_HIVE_CLOUDERA_JARS and ZJDBC_IMPALA_CLOUDERA_JARS refer to the folders in which Hive JDBC drivers and Impala JDBC drivers provided by Cloudera have been placed in section JDBC Drivers.
...
The table can be maintained via transaction SM30.
Sample entry:
Entries explained:
...
Table entries can be if needed, maintained via transaction.
Table example:
/DVD/JAVA_CONFIG – stores parameters for Datavard Java connectors. The table needs to be populated with entries via transaction SE16.
Sample configuration:
Prerequisites:
...
Use report /DVD/XOR_GEN for this purpose.
Storage Management setup
...
/DVD/SM_SETUP > [Edit mode] > [New storage]
Entries explained:
...
The Hive metastore storage is created in a very similar way to the process of setting up the HDFS storage, but the values are different:
Entries explained:
...
In case of any issues with Java connector, Java logs can be read from an application server to determine source of issue with report /DVD/SM_HIVE_DISPLAY_JAVA_LOG.
Entries explained:
...