Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3698

Explore unable to get delegation token while fetching next results on HDP2.0

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: Explore
    • Labels:
      None
    • Rank:
      1|hzyzyf:

      Description

      Some queries succeed in this environment. However, the get next results fails with the following exception:

      2015-09-17 00:32:42,591 - ERROR [executor-15:c.c.c.e.e.QueryExecutorHttpHandler@158] - Got exception:
      java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
              at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.fetchNextResults(BaseHiveExploreService.java:841) ~[co.cask.cdap.cdap-explore-3.2.0-SNAPSHOT.jar:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.nextResults(BaseHiveExploreService.java:808) ~[co.cask.cdap.cdap-explore-3.2.0-SNAPSHOT.jar:na]
              at co.cask.cdap.explore.executor.QueryExecutorHttpHandler.getQueryNextResults(QueryExecutorHttpHandler.java:145) ~[co.cask.cdap.cdap-explore-3.2.0-SNAPSHOT.jar:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:79) [co.cask.http.netty-http-0.12.0.jar:na]
              at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:41) [co.cask.http.netty-http-0.12.0.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [io.netty.netty-3.6.6.Final.jar:na]
              at co.cask.cdap.common.http.AuthenticationChannelHandler.messageReceived(AuthenticationChannelHandler.java:59) [co.cask.cdap.cdap-common-3.2.0-SNAPSHOT.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
      Caused by: java.lang.reflect.InvocationTargetException: null
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at co.cask.cdap.explore.service.hive.Hive12ExploreService.doFetchNextResults(Hive12ExploreService.java:97) ~[co.cask.cdap.cdap-explore-3.2.0-SNAPSHOT.jar:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.fetchNextResults(BaseHiveExploreService.java:836) ~[co.cask.cdap.cdap-explore-3.2.0-SNAPSHOT.jar:na]
              ... 22 common frames omitted
      Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authentication
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5895)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:833)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59648)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
      
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:239) ~[hive-service-0.12.0.2.0.13.0-43.jar-1442449693866.jar:0.12.0.2.0.13.0-43]
              at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170) ~[hive-service-0.12.0.2.0.13.0-43.jar-1442449693866.jar:0.12.0.2.0.13.0-43]
              at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:432) ~[hive-service-0.12.0.2.0.13.0-43.jar-1442449693866.jar:0.12.0.2.0.13.0-43]
              at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:306) ~[hive-service-0.12.0.2.0.13.0-43.jar-1442449693866.jar:0.12.0.2.0.13.0-43]
              ... 28 common frames omitted
      Caused by: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authentication
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5895)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:833)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59648)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
      
              at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:551) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:489) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1494) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:217) ~[hive-service-0.12.0.2.0.13.0-43.jar-1442449693866.jar:0.12.0.2.0.13.0-43]
              ... 31 common frames omitted
      Caused by: org.apache.hadoop.ipc.RemoteException: Delegation Token can be issued only with kerberos or web authentication
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5895)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:833)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59648)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.ipc.Client.call(Client.java:1300) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at com.sun.proxy.$Proxy43.getDelegationToken(Unknown Source) ~[na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at com.sun.proxy.$Proxy43.getDelegationToken(Unknown Source) ~[na:na]
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:805) ~[hadoop-hdfs-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:847) ~[hadoop-hdfs-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1318) ~[hadoop-hdfs-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:526) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:504) ~[hadoop-common-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121) ~[hadoop-mapreduce-client-core-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) ~[hadoop-mapreduce-client-core-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) ~[hadoop-mapreduce-client-core-2.2.0.2.0.13.0-43.jar:na]
              at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:381) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:515) ~[hive-exec-0.12.0.2.0.13.0-43.jar:0.12.0.2.0.13.0-43]
              ... 35 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bhooshan Bhooshan Mogal
                Reporter:
                bhooshan Bhooshan Mogal
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: