Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11704

Hive ATSHook fails to execute in Secure Hadoop clusters

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.1
    • Fix Version/s: 4.3.0
    • Component/s: Explore
    • Labels:
    • Release Notes:
      Fixed CDAP to work with and publish to Yarn Timeline Server in a secure environment.
    • Rank:
      1|i002lr:

      Description

      When we execute explore query in a secure Hadoop cluster on HDP 2.6.x, it fails with the following exception. This is because HDP 2.6.x has Hive ATSHook enabled by default, and CDAP Master does not pass the delegation token from the Timeline server to the Explore container.

      2017-05-12 18:43:31,204 - ERROR [HiveServer2-Background-Pool: Thread-143:o.a.h.h.q.e.Task@220] - Failed to execute tez graph.
      java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientRetryOpForOperateDelegationToken.run(TimelineClientImpl.java:704) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:186) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:465) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:375) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:359) ~[hadoop-yarn-client-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330) ~[hadoop-yarn-client-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250) ~[hadoop-yarn-client-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) ~[tez-api-0.7.0.2.6.0.3-8.jar:0.7.0.2.6.0.3-8]
      	at org.apache.tez.client.TezClient.start(TezClient.java:383) ~[tez-api-0.7.0.2.6.0.3-8.jar:0.7.0.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:197) ~[1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:279) ~[1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:159) ~[1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1748) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1494) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1291) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1153) [1494554715299-hive-exec-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197) [1494554719352-hive-service-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76) [1494554719352-hive-service-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253) [1494554719352-hive-service-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_112]
      	at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_112]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) [hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264) [1494554719352-hive-service-1.2.1000.2.6.0.3-8.jar:1.2.1000.2.6.0.3-8]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:333) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$1.run(TimelineClientImpl.java:371) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$1.run(TimelineClientImpl.java:363) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_112]
      	at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_112]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) [hadoop-common-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientRetryOpForOperateDelegationToken.run(TimelineClientImpl.java:702) ~[hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:na]
      	... 32 common frames omitted
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.8.0_112]
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[na:1.8.0_112]
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.8.0_112]
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[na:1.8.0_112]
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.8.0_112]
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.8.0_112]
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:309) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_112]
      	at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_112]
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:285) ~[hadoop-auth-2.7.3.2.6.0.3-8.jar:na]
      	... 44 common frames omitted
      2017-05-12 18:43:31,210 - WARN  [HiveServer2-Background-Pool: Thread-143:E.stderr@91] - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
      2017-05-12 18:43:31,210 - ERROR [HiveServer2-Background-Pool: Thread-143:o.a.h.h.q.Driver@993] - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: