Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10398

Error while using Spark ML plugin

    XMLWordPrintableJSON

    Details

    • Rank:
      1|hzzakv:

      Description

      While using the spark ML plugin, we are seeing the following error. This used to work before the latest spark changes on the release branch

      016-04-08 12:06:30,647 - INFO  [pcontroller-program:default.Training8.workflow.DataPipelineWorkflow-f8622a1a-fdbc-11e5-b5ca-323c28972b5d:c.c.c.i.a.s.ProgramLifecycleService$1@343] - Program stopped with error program:default.Training8.workflow.DataPipelineWorkflow, f8622a1a-fdbc-11e5-b5ca-323c28972b5d
      java.lang.AbstractMethodError: co.cask.cdap.etl.api.batch.SparkSink.run(Lco/cask/cdap/etl/api/batch/SparkExecutionPluginContext;Lorg/apache/spark/api/java/JavaRDD;)V
      	at co.cask.cdap.etl.batch.spark.ETLSparkProgram.run(ETLSparkProgram.java:128) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkTransactional$2.run(SparkTransactional.java:232) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkTransactional.executeLong(SparkTransactional.java:206) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:104) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext.execute(DefaultSparkExecutionContext.scala:140) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.DefaultJavaSparkExecutionContext.execute(DefaultJavaSparkExecutionContext.scala:72) ~[na:na]
      	at co.cask.cdap.etl.batch.spark.ETLSparkProgram.run(ETLSparkProgram.java:82) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:48) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala) ~[na:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_79]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_79]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
      	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) ~[na:na]
      	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) ~[na:na]
      	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) ~[na:na]
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) ~[na:na]
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:167) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:113) ~[na:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
      	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
      2016-04-08 12:06:32,478 - DEBUG [delay-close-thread-program:default.Training8.spark.phase-2-feb1c3dc-fdbc-11e5-8f8e-323c28972b5d:c.c.c.a.r.s.SparkProgramRunner$2$1@249] - Closed ClassLoader for SparkProgramRunner of program:default.Training8.spark.phase-2, run feb1c3dc-fdbc-11e5-8f8e-323c28972b5d.
      2016-04-08 12:06:50,367 - DEBUG [pool-2-thread-1:c.c.c.i.a.s.ProgramLifecycleService$RunRecordsCorrectorRunnable@818] - Start correcting invalid run records ...
      2016-04-08 12:06:50,371 - DEBUG [pool-2-thread-1:c.c.c.i.a.s.ProgramLifecycleService$RunRecordsCorrectorRunnable@823] - End correcting invalid run records.
      2016-04-08 12:06:54,561 - DEBUG [tx-snapshot:c.c.t.TransactionManager@398] - Starting snapshot of transaction state with timestamp 1460142414561
      2016-04-08 12:06:54,561 - DEBUG [tx-snapshot:c.c.t.TransactionManager@399] - Returning snapshot of state: TransactionSnapshot{timestamp=1460142414561, readPointer=1460142414046000000, writePointer=1460142414046000000, invalidSize=4, inProgressSize=1, committingSize=0, committedSize=0}
      
      
      

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: