To keep Hadoop connection in Storage management in operational status, periodic maintenance should be performed. This page discusses various events that can arise in the productive environment after the initial setup, and their resolution.

Kerberos keytab expiration

User principals used to authenticate against a Hadoop cluster secured with kerberos usually have limited validity (for example 1 year). Validity of the keytab should be checked after the initial setup and noted down to prevent an unnecessary downtime of the connection. 

Symptoms

 Extractions processed which previously worked are failing. Storage check in /dvd/sm_setup is failing. In Java logs error messages mentioning keytab out of validity period, or error authenticating user from keytab appear without changes of the setup.

Solution

Team that is responsible for the user maintenance needs to unlock and set a new password for the technical user used for connection from SAP to Hadoop. After the user is valid again, a new keytab must be exported from the KDC. This keytab needs to replace the existing one stored on the SAP system - usually $DIR_GLOBAL/security/dvd_conn/<SID>.keytab. Name of the keytab file should be the same as the old one to avoid changes in the Storage management setup.

SSL certificate expiration

Server certificates used for encrypted communication to Hadoop services usually have limited validity. When the validity is reached, the certificates are regenerated on the Hadoop cluster.

Symptoms

After this validity is reached, scheduled replications will fail as well as a storage check in /dvd/sm_setup. Logs in Datavard java application display errors mentioning failure to establish trust, incomplete certificate chain, or other SSL errors. SAP GUI should open a pop-up by every login a week before a certificate is reaching the end of it's validity to alert the users.

Solution

After the SSL certificates are regenerated on the Hadoop side, new trust needs to be established on the Storage Management (SM) side. Since SM uses two interfaces (SAP HTTP RFC and Java), expired certificates must to be replaced in STRUST and in Java truststore usually stored at $DIR_GLOBAL/security/dvd_conn/jssecacerts. Please follow the Hadoop storage setup guide for details on securing the connection with SSL.

Addition of new SAP application server

After a new SAP AS ABAP server is added to the SAP system, it will not be able to execute replication jobs to Hadoop in most environments. 

Symptoms

After the AS ABAP is added, scheduled jobs that should start on this AS fail, while jobs on other AS finish successfully. When an administrator changes this AS using SM51, the storage check in /dvd/sm_setup fails, while on other AS it runs correctly. 

Solution

Storage Management setup needs to be performed on this application server. Make sure the following points were executed on this new application server:

• Datavard specific folders that must exist were created as they do on other application servers
• System has Java installed on the same path as other application servers
• SAP Java connector (SAP JCO, libsapjco3.so) is installed on the application server and $LD_LIBRARY_PATH of <SID>adm user leads to this library
• If kerberos is used, ict/disable_cookie_urlencoding is set to '1' or '2'

Please refer to the installation guide for details on the steps mentioned above.