Uploaded image for project: 'CDAP Plugins'
  1. CDAP Plugins
  2. PLUGIN-215

Deployment of Spark Scala Program plugin fails

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 6.3.0
    • Component/s: None
    • Sprint:
      Cirus Sprint 1

      Description

      Deployment of Spark Scala Program plugin fails. Attached is the screenshot.

      steps to reproduce:

      Deploy dynamic spark plugin from hub and click on ScalaSparkProgram action plugin.

      2019-10-22 18:14:42,420 - ERROR [appfabric-executor-100:i.c.c.g.h.AppLifecycleHttpHandler$1@414] - Deploy failure
      java.lang.IllegalArgumentException: Failed to configure pipeline: Stage 'ScalaSparkProgram' encountered : io.cdap.cdap.api.spark.dynamic.CompilationFailureException: <source-string-1>:1: error: object cdap is not a member of package co.cask
      import co.cask.cdap.api.spark._
                     ^
      <source-string-1>:5: error: not found: type SparkMain
      class SparkProgram extends SparkMain {
                                 ^
      <source-string-1>:8: error: not found: type SparkExecutionContext
        override def run(implicit sec: SparkExecutionContext): Unit = {
                                       ^
              at io.cdap.cdap.datapipeline.SmartWorkflow.configure(SmartWorkflow.java:177) ~[na:na]
              at io.cdap.cdap.api.workflow.AbstractWorkflow.configure(AbstractWorkflow.java:58) ~[na:na]
              at io.cdap.cdap.app.DefaultAppConfigurer.addWorkflow(DefaultAppConfigurer.java:173) ~[na:na]
              at io.cdap.cdap.api.app.AbstractApplication.addWorkflow(AbstractApplication.java:118) ~[na:na]
              at io.cdap.cdap.datapipeline.DataPipelineApp.configure(DataPipelineApp.java:66) ~[na:na]
              at io.cdap.cdap.api.app.AbstractApplication.configure(AbstractApplication.java:59) ~[na:na]
              at io.cdap.cdap.internal.app.deploy.InMemoryConfigurator.createResponse(InMemoryConfigurator.java:155) ~[na:na]
              at io.cdap.cdap.internal.app.deploy.InMemoryConfigurator.config(InMemoryConfigurator.java:116) ~[na:na]
              at io.cdap.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:110) ~[na:na]
              at io.cdap.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:57) ~[na:na]
              at io.cdap.cdap.pipeline.AbstractStage.process(AbstractStage.java:53) ~[na:na]
              at io.cdap.cdap.internal.pipeline.SynchronousPipeline.execute(SynchronousPipeline.java:57) ~[na:na]
              at io.cdap.cdap.internal.app.deploy.LocalApplicationManager.deploy(LocalApplicationManager.java:132) ~[na:na]
              at io.cdap.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:668) ~[na:na]
              at io.cdap.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:471) ~[na:na]
              at io.cdap.cdap.gateway.handlers.AppLifecycleHttpHandler$1.onFinish(AppLifecycleHttpHandler.java:390) ~[na:na]
              at io.cdap.cdap.common.http.AbstractBodyConsumer.finished(AbstractBodyConsumer.java:65) [na:na]
              at io.cdap.http.internal.HttpMethodInfo.bodyConsumerFinish(HttpMethodInfo.java:151) [io.cdap.http.netty-http-1.3.1.jar:na]
              at io.cdap.http.internal.HttpMethodInfo.chunk(HttpMethodInfo.java:112) [io.cdap.http.netty-http-1.3.1.jar:na]
              at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:47) [io.cdap.http.netty-http-1.3.1.jar:na]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) [io.cdap.http.netty-http-1.3.1.jar:na]
              at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_221]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_221]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_221]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_221]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_221]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_221]
              at java.lang.Thread.run(Thread.java:748) [na:1.8.0_221]
      Caused by: io.cdap.cdap.etl.api.validation.ValidationException: Errors were encountered during validation.
              at io.cdap.cdap.etl.validation.DefaultFailureCollector.getOrThrowException(DefaultFailureCollector.java:77) ~[na:na]
              at io.cdap.cdap.etl.spec.PipelineSpecGenerator.getPlugin(PipelineSpecGenerator.java:381) ~[na:na]
              at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:278) ~[na:na]
              at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:211) ~[na:na]
              at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStages(PipelineSpecGenerator.java:159) ~[na:na]
              at io.cdap.cdap.etl.batch.BatchPipelineSpecGenerator.generateSpec(BatchPipelineSpecGenerator.java:57) ~[na:na]
              at io.cdap.cdap.datapipeline.SmartWorkflow.configure(SmartWorkflow.java:174) ~[na:na]
              ... 32 common frames omitted 

        Attachments

          Activity

            People

            • Assignee:
              vjosadika Vjosa Dika
              Reporter:
              vinisha Vinisha Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: