Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6206

If Spark is not available, deploying a Spark application fails with a misleading error message

    Details

    • Release Notes:
      Improve error message on application deployment failure due to missing Spark library
    • Rank:
      1|hzzefz:

      Description

      My Spark home in my cluster is not configured. Therefore, deploying an app with Spark (including a data pipeline that runs in spark) fails. But it fails with a misleading error message:

      Upload failed co.cask.cdap.api.spark.SparkMain
      

      and the logs contain this stack trace:

      2016-06-15 21:48:31,079 - ERROR [netty-executor-1828:c.c.c.c.HttpExceptionHandler@49] - Unexpected error: request=PUT /v3/namespaces/default/apps/Stream2TPFSAvro user=<null>:
      java.lang.NoClassDefFoundError: co/cask/cdap/api/spark/JavaSparkMain
              at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_75]
              at java.lang.ClassLoader.defineClass(ClassLoader.java:800) ~[na:1.7.0_75]
              at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.7.0_75]
              at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) ~[na:1.7.0_75]
              at java.net.URLClassLoader.access$100(URLClassLoader.java:71) ~[na:1.7.0_75]
              at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.7.0_75]
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_75]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_75]
              at co.cask.cdap.etl.batch.spark.ETLSpark.configure(ETLSpark.java:75) ~[na:na]
              at co.cask.cdap.api.spark.AbstractSpark.configure(AbstractSpark.java:36) ~[co.cask.cdap.cdap-api-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.app.DefaultAppConfigurer.addSpark(DefaultAppConfigurer.java:134) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.datapipeline.SmartWorkflow.addProgram(SmartWorkflow.java:253) ~[na:na]
              at co.cask.cdap.datapipeline.SmartWorkflow.configure(SmartWorkflow.java:105) ~[na:na]
              at co.cask.cdap.api.workflow.AbstractWorkflow.configure(AbstractWorkflow.java:55) ~[co.cask.cdap.cdap-api-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.app.DefaultAppConfigurer.addWorkflow(DefaultAppConfigurer.java:146) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.api.app.AbstractApplication.addWorkflow(AbstractApplication.java:126) ~[co.cask.cdap.cdap-api-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.datapipeline.DataPipelineApp.configure(DataPipelineApp.java:78) ~[na:na]
              at co.cask.cdap.api.app.AbstractApplication.configure(AbstractApplication.java:61) ~[co.cask.cdap.cdap-api-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.getSpecJson(InMemoryConfigurator.java:184) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.createResponse(InMemoryConfigurator.java:154) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.config(InMemoryConfigurator.java:128) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:82) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:46) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.pipeline.AbstractStage.process(AbstractStage.java:52) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.pipeline.SynchronousPipeline.execute(SynchronousPipeline.java:56) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.deploy.LocalApplicationManager.deploy(LocalApplicationManager.java:119) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:506) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:343) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.gateway.handlers.AppLifecycleHttpHandler$1.onFinish(AppLifecycleHttpHandler.java:282) ~[co.cask.cdap.cdap-app-fabric-3.5.0-SNAPSHOT.jar:na]
              at co.cask.cdap.common.http.AbstractBodyConsumer.finished(AbstractBodyConsumer.java:65) ~[co.cask.cdap.cdap-common-3.5.0-SNAPSHOT.jar:na]
              at co.cask.http.HttpMethodInfo.bodyConsumerFinish(HttpMethodInfo.java:149) [co.cask.http.netty-http-0.14.0.jar:na]
              at co.cask.http.HttpMethodInfo.chunk(HttpMethodInfo.java:108) [co.cask.http.netty-http-0.14.0.jar:na]
              at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:40) [co.cask.http.netty-http-0.14.0.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
      Caused by: java.lang.ClassNotFoundException: co.cask.cdap.api.spark.JavaSparkMain
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_75]
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]
      

      There is no indication that this is because SPARK_HOME is not set. We should have a better error message.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: