Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11605

NoSuchMethodException for "sparkprogram"

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|i0021b:

      Description

      1. Created pipeline by using "sparkprogram" plug-in on Azure environment using CDAP4.1.1.

      @Plugin(type = "sparkprogram")
      @Name("DecompStep1Spark")
      public class DecompStep1Spark implements JavaSparkMain {
      private static final Logger LOG = LoggerFactory.getLogger(DecompStep1Spark.class);

      @Override
      public void run (JavaSparkExecutionContext sec) throws Exception

      { //code logic }

      }

      2. mvn clean package to generate jar file test.jar.
      3. manually create the matched json file test.json:
      {
      "parents": [
      "system:cdap-data-pipeline[4.1.0-SNAPSHOT,4.3.0-SNAPSHOT)"
      ],
      "properties":

      { "doc.Test": "Test transform Actions." }

      }

      4. "Add Entity" -> Upload "Plugin"
      5. "DecompStep1Spark" will be shown in "Action" group.
      6. Use "DecompStep1Spark" to create a pipeline.

      The exception was thrown when run the pipeline:

      java.util.concurrent.ExecutionException: java.lang.NoSuchMethodException: co.cask.cdap.app.spark.DecompStep1Spark.main([Ljava.lang.String
      at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
      at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:277) ~[co.cask.cdap.cdap-spark-core-4.1.1.jar:na]
      at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.app.runtime.spark.SparkRuntimeService$2$1.run(SparkRuntimeService.java:335) [co.cask.cdap.cdap-spark-core-4.1.1.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
      Caused by: java.lang.NoSuchMethodException: co.cask.cdap.app.spark.DecompStep1Spark.main([Ljava.lang.String
      at java.lang.Class.getMethod(Class.java:1786) ~[na:1.8.0_121]
      at co.cask.cdap.datapipeline.JavaSparkMainWrapper.run(JavaSparkMainWrapper.java:52) ~[na:na]
      at co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:104) ~[na:na]
      at co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala) ~[na:na]
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
      at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) ~[na:na]
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) ~[na:na]
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) ~[na:na]
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) ~[na:na]
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[na:na]
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:168) ~[na:na]
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:55) ~[na:na]
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:112) ~[na:na]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121]
      ... 1 common frames omitted

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              yan yan stein
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: