(SM-1905) Troubleshooting and Error Log

Owned by SNP
May 06, 2019

Purpose of this page is to provide a reference for errors and their possible resolution, that user can encounter while using Hadoop storage.

SAP errors



ICM_HTTP_INTERNAL_ERROR

Generic ICM error. Check DEV_ICM logs in ST11 for details. 

  • In this example RFC is set SSL inactive, but HttpFS service is SSL enabled. Set the RFC as SSL active.
  • In this example, RFC is set as SSL active, but there is no correct server or CA certificate stored in STRUST, therefore trust can't be established. Add correct certificate to STRUST.

ICM_HTTP_CONNECTION_FAILED

Generic ICM error. Check DEV_ICM logs in ST11 for details. 

  • In this example, service is not listening on the port specified. Make sure that HttpFS service is available on that host and port and there is are no blocked ports.

SSSLERR_NO_SSL_RESPONSE 

HttpFS RFC in SM69 has checkbox SSL set as Active, while the HttpFS service itself is not SSL secured. Set checkbox to inactive. 

No HDFS host available for connection

Can be found in the job log. Make sure that HTTP/HTTPS service is correctly set in ICM and active. Go to t-code SMICM → Go to → Services and make sure both HTTP and HTTPS have service name/port maintained.

SAP OS command 'xx' is different than expected

Datavard generates logical commands in SM69, based on parameters provided in /DVD/JCO_MNG. If path to Java executable is changed in this transaction, it no longer matches the generated logical command.

Make sure that path to Java exe is in sync between /DVD/JCO_MNG and SM69.

Enter logon data pop-up in Storage management

Make sure that parameter ict/disable_cookie_urlencoding is set to '1' (or '2' on latest releases). This happens because without this parameter SAP sends incorrect authentication cookie and receives response with 401 Unauthorized.

Java errors 

Datavard Java connector is used for authentication with Kerberos, and for executing SQL queries.

Logs from Datavard java connectors can be displayed either in transaction /DVD/JCO_MNG or by running report /DVD/SM_HIVE_DISPLAY_JAVA_LOG. 


Authentication errors

Java authentication errors can be very non-descriptive and differ between two Java vendors. This section provides couple of examples and their resolutions.

https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/errors.html is also very good source for troubleshooting.

Login failure for 'xx' from keytab 'xx' - connection timed out

Most likely KDC is unreachable. Try telnet to port 88 on the KDC. 

java.io.IOException: Login failure for dc1@SOFT.FAU from keytab /sapmnt/DC1/global/security/dvd_conn/dc1.keytab: javax.security.auth.login.FailedLoginException: Login error: java.net.SocketTimeoutException: connect timed out
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) ~[auth_conn.jar:?]
at com.datavard.http.authentication.KerberosAuthenticator.login(KerberosAuthenticator.java:75) ~[auth_conn.jar:?]
at com.datavard.http.authentication.KerberosAuthenticator.executeRequest(KerberosAuthenticator.java:38) ~[auth_conn.jar:?]
at com.datavard.http.authentication.DVDHttpDAO.send(DVDHttpDAO.java:24) [auth_conn.jar:?]
at com.datavard.http.handler.DVDExecuteHttpRequestHandler.handleRequest(DVDExecuteHttpRequestHandler.java:37) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker$FunctionDispatcher.handleRequest(DefaultServerWorker.java:1036) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker$FunctionDispatcher.handleRequest(DefaultServerWorker.java:972) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker.dispatchRequest(DefaultServerWorker.java:148) [auth_conn.jar:?]
at com.sap.conn.jco.rt.MiddlewareJavaRfc$JavaRfcServer.dispatchRequest(MiddlewareJavaRfc.java:3415) [auth_conn.jar:?]
at com.sap.conn.jco.rt.MiddlewareJavaRfc$JavaRfcServer.listen(MiddlewareJavaRfc.java:2468) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker.dispatch(DefaultServerWorker.java:254) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker.loop(DefaultServerWorker.java:346) [auth_conn.jar:?]
at com.sap.conn.jco.rt.DefaultServerWorker.run(DefaultServerWorker.java:232) [auth_conn.jar:?]
at java.lang.Thread.run(Thread.java:811) [?:2.9 (12-15-2017)]
Caused by: javax.security.auth.login.FailedLoginException: Login error: java.net.SocketTimeoutException: connect timed out


Peer indicated failure: Error validating the login

Usually appears when username + password is used for authentication (AuthMech=3). Make sure that password is correct and hashed with /DVD/XOR_GEN. Also make sure that correct authentication type is selected.

Unable to connect to server: Failure to initialize security context.

Either path to krb5.conf is incorrect, or krb5.conf is not configured correctly.

Error creating login context using ticket cache: Unable to obtain Principal name for authentication.

Very general error. Usually refers to inconsistency of information stored in .jaas config file and reality. 

  • Check if path to keytab in .jaas is correct
  • Check if principal name in .jaas is correct

Unable to connect to server: GSS initiate failed

Very generic error. Can be an issue with kerbreros config file. Make sure that principal's domain is correctly set in this configuration file.

Error creating login context using ticket cache: Login Failure: all modules ignored.

Usually refers to incorrect .jaas file.

Example: You are running IBM java, but .jaas is generated for Oracle Java. 

Go to /DVD/JCO_MNG, set correct Java vendor, delete .jaas file from the OS level and click check storage again. Correct file should be generated.

Please note that .jaas file is being only generated when Java is not running when Check Storage is performed.

Error initialized or created transport for authentication: java.io.IOException 'xx' (No such file or directory)

.Jaas file is not generated. Stop java, make sure that Cloudera drivers config path parameter is maintained in the storage and click check storage. Jaas file should be generated.



Other errors

JDBC driver not initialized correctly. ClassNotFoundException

Path to JDBC driver stored in SAP is either incorrect, or classname of the driver provided is wrong. Check the logical paths and physical location of drivers, as well as driver documentation for correct classname.

Return code 2 while executing insert from select

Usually refers to a failed mapreduce job due to not enough resources.

Make sure that dynamic partitioning is set to nonstrict:

  • hive.exec.dynamic.partition.mode=nonstrict

Try adding following parameters to storage management field Hints for Hive/Impala (separated by ; ) to increase resources

  • mapred.child.java.opts=-Xmx4916m
  • hive.optimize.sort.dynamic.partition=true
  • hive.exec.max.dynamic.partitions=10000
  • hive.exec.max.dynamic.partitions.pernode=10000;


Name or password is incorrect

Java doesn't start, but logs name or password is incorrect error.

Refers to invalid username - password combination maintained in /DVD/JCO_MNG for the Java version. Make sure that correct password is provided and that the user is not locked.


String index out of range

Usually refers to a typo in password, or password that is not properly encrypted. All passwords (truststore password, hive password, impala password,...) need to be hashed using report /DVD/XOR_GEN

Make sure that all hashed passwords are correct in /DVD/SM_SETUP and table /DVD/HDP_CUS_C



User 'xx' does not have privileges to access 'xx'

Sentry/Ranger permissions are not correctly set. Refer to the installation guide on how to set them and correct the setup.


Certificate chaining error

System can't establish secure TLS encrypted session. Make sure that correct certificates are stored in Jssecacerts file (java truststore).


Error during closing of session - Session <ID> does not exist

Happens when 2 java processes with same program ID are running on the same application server. Kill one of the processes. This issue is fixed in latest versions of Java connector.


Connection timeout expired

Usually caused by unavailability of target service, or network instability.

java.sql.SQLException: [Cloudera][HiveJDBCDriver](700100) Connection timeout expired. Details: None.