Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-8636

Spark programs fail to be submitted in CDH 5.8.4

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.0
    • Fix Version/s: 4.1.0
    • Component/s: Spark
    • Labels:
      None
    • Release Notes:
      Fixed an issue with Spark programs not working against CDH 5.8.4.
    • Rank:
      1|hzzvqv:

      Description

      I attempted to launch a Spark program and it fails with the following error.
      Spark programs work against CDH 5.8.3, but fail against CDH 5.8.4.

      I suspect the regression is related to the change brought in by the following JIRA:
      https://issues.apache.org/jira/browse/SPARK-12241
      https://github.com/apache/spark/pull/10227
      (Source: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_58.html#fixed_issues584)

      2017-02-20T19:23:57,988Z ERROR c.c.c.i.a.r.d.AbstractProgramTwillRunnable [<HOSTNAME>] [TwillContainerService] AbstractProgramTwillRunnable:run(Abstrac
      tProgramTwillRunnable.java:329) - Program DataStreamsSparkStreaming execution failed.
      java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.Access
      DeniedException: Token generation only allowed for Kerberos authenticated clients
              at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:123)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$1.getAuthenticationToken(AuthenticationProtos.java:4267)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService.callMethod(AuthenticationProtos.java:4387)
              at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7777)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
              at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
              at java.lang.Thread.run(Thread.java:745)
      
              at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294)
              at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281)
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:324)
              at org.apache.twill.internal.container.TwillContainerService.doRun(TwillContainerService.java:221)
              at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:189)
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
              at java.lang.Thread.run(Thread.java:745)
      
      Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
              at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:123)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$1.getAuthenticationToken(AuthenticationProtos.java:4267)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService.callMethod(AuthenticationProtos.java:4387)
              at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7777)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
              at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
              at java.lang.Thread.run(Thread.java:745)
              at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294)
              at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281)
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
              at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:273)
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
              at co.cask.cdap.app.runtime.spark.SparkRuntimeService$2$1.run(SparkRuntimeService.java:331)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
              at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:123)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$1.getAuthenticationToken(AuthenticationProtos.java:4267)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService.callMethod(AuthenticationProtos.java:4387)
              at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7777)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
              at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
              at java.lang.Thread.run(Thread.java:745)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
              at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1631)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
              at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
              at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBaseInner(YarnSparkHadoopUtil.scala:305)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBase(YarnSparkHadoopUtil.scala:278)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBase(YarnSparkHadoopUtil.scala:160)
              at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:360)
              at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:732)
              at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:142)
              at org.apache.spark.deploy.yarn.Client.run(Client.scala:1028)
              at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1088)
              at org.apache.spark.deploy.yarn.Client.main(Client.scala)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
              at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
              at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:168)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:55)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:112)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
              at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:123)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$1.getAuthenticationToken(AuthenticationProtos.java:4267)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService.callMethod(AuthenticationProtos.java:4387)
              at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7777)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
              at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
              at java.lang.Thread.run(Thread.java:745)
      
              at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1268)
              at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
              at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
              at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
              at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
              at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBaseInner(YarnSparkHadoopUtil.scala:305)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBase(YarnSparkHadoopUtil.scala:278)
              at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHBase(YarnSparkHadoopUtil.scala:160)
              at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:360)
              at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:732)
              at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:142)
              at org.apache.spark.deploy.yarn.Client.run(Client.scala:1028)
              at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1088)
              at org.apache.spark.deploy.yarn.Client.main(Client.scala)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
              at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
              at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:168)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:55)
              at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:112)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: