To keep Hadoop connection in Storage management in operational status, periodic maintenance should be performed. This page discusses various events that can arise in the productive environment after the initial setup, and their resolution.

Kerberos keytab expiration

User principals used to authenticate against a Hadoop cluster secured with kerberos usually have limited validity (for example 1 year). Validity of the keytab should be checked after the initial setup and noted down to prevent an unnecessary downtime of the connection. 

Symptoms

 Extractions processed which previously worked are failing. Storage check in /dvd/sm_setup is failing. In Java logs error messages mentioning keytab out of validity period, or error authenticating user from keytab appear without changes of the setup.

Solution

Team that is responsible for the user maintenance needs to unlock and set a new password for the technical user used for connection from SAP to Hadoop. After the user is valid again, a new keytab must be exported from the KDC. This keytab needs to replace the existing one stored on the SAP system - usually $DIR_GLOBAL/security/dvd_conn/<SID>.keytab. Name of the keytab file should be the same as the old one to avoid changes in the Storage management setup.

SSL certificate expiration

Server certificates used for encrypted communication to Hadoop services usually have limited validity. When the validity is reached, the certificates are regenerated on the Hadoop cluster.

Symptoms

After this validity is reached, scheduled replications will fail as well as a storage check in /dvd/sm_setup. Logs in Datavard java application display errors mentioning failure to establish trust, incomplete certificate chain, or other SSL errors. SAP GUI should open a pop-up by every login a week before a certificate is reaching the end of it's validity to alert the users.

Solution

After the SSL certificates are regenerated on the Hadoop side, new trust needs to be established on the Storage Management (SM) side. Since SM uses two interfaces (SAP HTTP RFC and Java), expired certificates must to be replaced in STRUST and in Java truststore usually stored at $DIR_GLOBAL/security/dvd_conn/jssecacerts. Please follow the Hadoop storage setup guide for details on securing the connection with SSL.

Addition of new SAP application server

After a new SAP AS ABAP server is added to the SAP system, it will not be able to execute replication jobs to Hadoop in most environments. 

Symptoms

After the AS ABAP is added, scheduled jobs that should start on this AS fail, while jobs on other AS finish successfully. When an administrator changes this AS using SM51, the storage check in /dvd/sm_setup fails, while on other AS it runs correctly. 

Solution

Storage Management setup needs to be performed on this application server. Make sure the following points were executed on this new application server:

• Datavard specific folders that must exist were created as they do on other application servers
• System has Java installed on the same path as other application servers
• SAP Java connector (SAP JCO, libsapjco3.so) is installed on the application server and $LD_LIBRARY_PATH of <SID>adm user leads to this library
• If kerberos is used, ict/disable_cookie_urlencoding is set to '1' or '2'

Please refer to the installation guide for details on the steps mentioned above.

Refresh of SAP system

Quality and Development SAP systems are usually periodically refreshed as a copy of production to provide a better testing environment. This will cause inconsistency between metadata used by Datavard products (PROD) and data stored in on Hadoop side (QUALITY). Also Storage Management on the refreshed system will be pointing to the Hadoop environment of production. 

To maintain consistency of metadata used by Datavard products and data stored on Hadoop, 3 manual actions need to be performed:

• Copy data from Production Hadoop Hive database to Quality Hive database
• Create new connection in Quality SAP system pointing to Quality Hadoop environment
• Datavard metadata correction

Copying data from Production Hadoop to Quality

1. Create a backup of quality Hive database and HDFS landing folder
2. Load Hive data from production database to quality database. Recommended is to use Hive Import/Export function https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
3. Copy content of HDFS landing folder from Production to Quality. Recommended is to use discp

Storage recreation

Since Storage management still points to production environment after the system copy, a new storage needs to be created to point to the Quality Hadoop environment. Usually after the copy some configuration steps (mainly on OS level of SAP) are lost and need to be created from scratch. 

Please follow Hadoop storage setup guide to establish a new connection to Hadoop. Make sure that all information stored in SAP database is changed to quality environment (usernames, hostnames, naming conventions). 

Datavard metadata correction

After this new storage ID is created, it should have different name than the one used in production. If this the case, metadata of structures tied to this ID needs to be fixed (changed from Prod storage ID to Quality storage ID), please contact Datavard customer support to assist you with this repair.