Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5350

Mapreduce fails on clusters with HDFS encryption

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.2
    • Fix Version/s: 3.4.0, 3.3.3
    • Component/s: MapReduce
    • Labels:
      None
    • Release Notes:
      Fixes an issue that prevents MapReduce to run on clusters with HDFS encryption.
    • Rank:
      1|hzz98v:

      Description

      After enabling HDFS Data at Rest Encryption using Cloudera Manager and a Java KMS, Mapreduce programs appear to fail. The "cdap" user is the KMS admin, as are users in the "sysadmin" group.

      2016-03-21 23:48:54,979 - INFO  [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@289] - Submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026
      2016-03-21 23:48:54,996 - INFO  [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.i.LocalClientProtocolProvider@42] - Using framework: yarn
      2016-03-21 23:48:55,659 - WARN  [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      2016-03-21 23:48:55,670 - WARN  [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:java.io.IOException: java.lang.reflect.UndeclaredThrowableException
      2016-03-21 23:48:55,688 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@300] - Exception when submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026
      java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      Caused by: java.lang.reflect.UndeclaredThrowableException: null
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 16 common frames omitted
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 17 common frames omitted
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67]
              at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              ... 28 common frames omitted
      2016-03-21 23:48:55,847 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception
      java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      Caused by: java.lang.reflect.UndeclaredThrowableException: null
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 16 common frames omitted
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 17 common frames omitted
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67]
              at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              ... 28 common frames omitted
      2016-03-21 23:48:55,879 - ERROR [pcontroller-program:default.PurchaseHistory.mapreduce.PurchaseHistoryBuilder-5424fb96-efbf-11e5-b900-42010af00026:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@311] - Program runner error out.
      java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      Caused by: java.lang.reflect.UndeclaredThrowableException: null
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 16 common frames omitted
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 17 common frames omitted
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67]
              at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              ... 28 common frames omitted
      2016-03-21 23:48:55,912 - ERROR [TwillContainerService:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@323] - Program PurchaseHistoryBuilder execution failed.
      java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:317) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at org.apache.twill.internal.container.TwillContainerService.doRun(TwillContainerService.java:130) [org.apache.twill.twill-yarn-0.6.0-incubating.jar:0.6.0-incubating]
              at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179) [org.apache.twill.twill-core-0.6.0-incubating.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na]
              ... 1 common frames omitted
      Caused by: java.lang.reflect.UndeclaredThrowableException: null
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 16 common frames omitted
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na]
              ... 17 common frames omitted
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67]
              at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67]
              at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na]
              ... 28 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              andreas Andreas Neumann
              Reporter:
              chris Chris Gianelloni
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: