Troubleshooting using Kerberos with Talend Big Data

For information on how to use Kerberos with Talend Big Data, see:

• How to use Kerberos in Talend Studio with Big Data v5.x (part 1)
• How to use Kerberos in Talend Studio with Big Data v6.x

When the debugger is enabled, additional information is captured which can help diagnose problems.To enable the debugger:

1. Navigate to the Run view of the Talend Studio.
2. Click to open the Advanced settings tab. Select the Use specific JVM arguments check box.
3. Click the New... button and add this argument:

-Dsun.security.krb5.debug=true

To collect debugging information, after enabling the debugger:

1. Navigate to the Basic Run tab.
2. Run the job.
3. Examine the collected log messages. In doing this, pay attention to the used principal and how the clien, that is to say, the Studio or your Hadoop cluster, is dealing with Kerberos from the cache.

Once the detailed error is identified, search for it in the errors below.

• "Exception in thread "main" java.lang.ExceptionInInitializerError
  
  at java.lang.Class.forName0(Native Method)
  
      at java.lang.Class.forName(Class.java:274)
      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2147)

  Possible Cause

  You java driver does not support the kerbberos security.

  Resolution

  Get http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html and unzip it on your $JAVAHOME/jre/lib/security.

• Caused by: java.lang.IllegalArgumentException: Illegal principal name user@BIGDATA.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user@BIGDATA.COM

  Possible Cause

  Possible realm conflict.

  Resolution

  Check your /etc/krb5.conf . If there are multiple realms, try setting your realm as the default one.

• GSSException: No valid credentials provided (Mechanism level: Connection refused: connect)

  Possible Cause

  In krb5.ini, the KDC hostname is incorrect or the KDC daemon is not started on this server.

  Resolution

  Check for the daemon status or ask to verify the right KDC.

• GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt

  Possible Cause

  User who executes the job has no valid ticket in his/her cache.

  Resolution

  Run kinit where the driver will run (on localhost if from studio, on the server where the JobServer lies if on remote).

• java.io.IOException: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/sandbox.hadoop.com@EXAMPLE.COM; Host Details : local host is: "ServerName1/127.0.1.1";

  Possible Cause

  The server name is not defined in /etc/hosts .

  Resolution

  Check etc/hosts on the client machine and make sure you have the same hostname.

  Or

  Replace _HOST by the server's complete hostname.

• java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN1/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Checksum failed

  Possible Cause

  The principal hash does not correspond to the hash the keytab knows. It can happen when the principal was once deleted and then recreated in the KDC database.

  Resolution

  Renew the obsolete keytab.

• java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: KrbException: Client not found in Kerberos database (devil) - CLIENT_NOT_FOUND

  Possible Cause

  The keytab correctly contains the user principal but KDC does not know this principal anymore.

  Resolution

  Renew the obsolete keytab.

• java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

  Possible Cause

  The keytab used does not contain any credentials corresponding to the specified principal or the keytab is not readable for the current user.

  Resolution

  Check the username specified in the Talend components, if ok, check the keytab privileges (read access) and validity.

• kinit: Client not found in Kerberos database while getting initial credentials

  Possible Cause

  The user has no principal in the KDC database.

  Resolution

  Create the principal or use the right one (via kadmin or kadmin.local).

• krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 1 KDC

  Possible Cause

  The KDC server is configured to use only UDP or TCP and not both, as supposed by your krb5.conf .

  Resolution

  Try to force the protocol in the krb5.conf by adding a line:

  kdc = tcp/<kdc_server_hostname>:88

• org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered coprocessor service found for name AuthenticationService in region hbase:meta,,1

  Possible Cause

  The HBase server side configurations for the coprocessor security are missing.

  Resolution

  Add the following to hbase-site.xml :

  <property> 
  	<name>hbase.coprocessor.region.classes</name> 
  	<value>org.apache.hadoop.hbase.security.token.TokenProvider,
  		   org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
  		   org.apache.hadoop.hbase.security.access.AccessController </value> 
  </property>
  <property> 
  	<name>hbase.rpc.engine</name> 
  	<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> 
  </property> 
  <property> 
  	<name>hbase.coprocessor.master.classes</name> 
  	<value>org.apache.hadoop.hbase.security.access.AccessController</value> 
  </property>

• org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): KERBEROS authentication is not enabled. Available:[SIMPLE]

  Possible Cause

  The Studio is configured to run a Job targeting a kerberized cluster whereas the server is not configured to use Kerberos.

  Resolution

  Remove the Kerberos credentials and configure the Job to access the cluster using simple authentication (user-based).

• org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

  Possible Cause

  Studio tries to connect to a non-kerberized cluster whereas this is a kerberized environment.

  Resolution

  Activate the Kerberos credentials in the Job / Talend components.

• org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

  Possible Cause

  Due to a bug of the studio for MapReduce Jobs, this error is happening when you activated Kerberos on a M/R Job but a username was previously set and the Studio still use it to access the cluster without taking Kerberos credentials into consideration.

  Resolution

  Deactivate the Kerberos authentication, set the username to blank and reactivate the Kerberos authentication.

• org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hdfs@TALEND.EXAMPLE.COM

  Possible Cause

  You face a cross-realm Kerberos environment and the mapping rules to translate principal from one realm to the Hadoop realm does not match the expected result.

  Resolution

  Check that the mapping rules are specified in the core-site.xml embedded in hadoop-conf-kerberos.jar . If yes, you have to correct these rules to obtain the right mapping.

• Server has invalid Kerberos principal: hdfs/talend-cdh5-nn1@TALEND.COM

  Possible Cause

  The realm configured in krb5.conf or the realm configured in the Job does not match the server realm.

  Resolution

  Check the Job configuration and the krb5.ini to make sure this is aligned with the target server realm.

• [WARN ]: org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:user@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for user@EXAMPLE.COM to hbase/talend-cdh5@EXAMPLE.COM

  Possible Cause

  The server name is not defined in /etc/hosts .

  Resolution

  Add FQDN to /etc/hosts .