Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-9958

Multiple sinks doesn't work with Spark engine

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.4.0
    • Component/s: Pipelines, Spark
    • Labels:
      None
    • Rank:
      1|hzzbrz:

      Description

      Attached a pipeline with elastic search sink and spark engine. It fails because the hadoop classes that elastic search plugin needs are not available.

      Stack trace:

      2016-04-23 19:33:35,902 - ERROR [Spark - phase-1:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception
      co.cask.tephra.TransactionFailureException: Failed to execute method co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit() inside a transaction
      	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:177) ~[co.cask.cdap.cdap-data-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService.beforeSubmit(SparkRuntimeService.java:316) ~[co.cask.cdap.cdap-spark-core-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService.startUp(SparkRuntimeService.java:137) ~[co.cask.cdap.cdap-spark-core-3.4.0-SNAPSHOT.jar:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$3$1.run(SparkRuntimeService.java:300) [co.cask.cdap.cdap-spark-core-3.4.0-SNAPSHOT.jar:na]
      	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
      Caused by: java.lang.UnsupportedOperationException: Hadoop Job is not available in Spark
      	at co.cask.cdap.etl.batch.spark.AbstractSparkBatchContext.getHadoopJob(AbstractSparkBatchContext.java:44) ~[na:na]
      	at co.cask.hydrator.plugin.batch.sink.BatchElasticsearchSink.prepareRun(BatchElasticsearchSink.java:72) ~[na:na]
      	at co.cask.hydrator.plugin.batch.sink.BatchElasticsearchSink.prepareRun(BatchElasticsearchSink.java:51) ~[na:na]
      	at co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit(ETLSpark.java:128) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$4.call(SparkRuntimeService.java:321) ~[co.cask.cdap.cdap-spark-core-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$4.call(SparkRuntimeService.java:316) ~[co.cask.cdap.cdap-spark-core-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:174) ~[co.cask.cdap.cdap-data-fabric-3.4.0-SNAPSHOT.jar:na]
      	... 5 common frames omitted
       2016-04-23 19:33:35,910 - ERROR [WorkflowDriver:c.c.c.i.a.r.w.WorkflowDriver@319] - Error executing the action phase-1 in the Workflow program_run:default.ContactLoaderSpark.workflow.DataPipelineWorkflow.f0855a1f-09c4-11e6-8716-ca445666d6f9.
      java.util.concurrent.ExecutionException: java.lang.RuntimeException: co.cask.tephra.TransactionFailureException: Failed to execute method co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit() inside a transaction
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_79]
      	at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_79]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.executeAction(WorkflowDriver.java:317) [co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.executeNode(WorkflowDriver.java:438) [co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.executeAll(WorkflowDriver.java:533) [co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver.run(WorkflowDriver.java:521) [co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na]
      	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
      Caused by: java.lang.RuntimeException: co.cask.tephra.TransactionFailureException: Failed to execute method co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit() inside a transaction
      	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.AbstractProgramWorkflowRunner$1.run(AbstractProgramWorkflowRunner.java:154) ~[co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.workflow.ProgramWorkflowAction.run(ProgramWorkflowAction.java:63) ~[co.cask.cdap.cdap-api-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver$1.call(WorkflowDriver.java:313) ~[co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.workflow.WorkflowDriver$1.call(WorkflowDriver.java:310) ~[co.cask.cdap.cdap-app-fabric-3.4.0-SNAPSHOT.jar:na]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_79]
      	... 1 common frames omitted
      Caused by: co.cask.tephra.TransactionFailureException: Failed to execute method co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit() inside a transaction
      	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:177) ~[co.cask.cdap.cdap-data-fabric-3.4.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService.beforeSubmit(SparkRuntimeService.java:316) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService.startUp(SparkRuntimeService.java:137) ~[na:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$3$1.run(SparkRuntimeService.java:300) ~[na:na]
      	... 1 common frames omitted
      Caused by: java.lang.UnsupportedOperationException: Hadoop Job is not available in Spark
      	at co.cask.cdap.etl.batch.spark.AbstractSparkBatchContext.getHadoopJob(AbstractSparkBatchContext.java:44) ~[na:na]
      	at co.cask.hydrator.plugin.batch.sink.BatchElasticsearchSink.prepareRun(BatchElasticsearchSink.java:72) ~[na:na]
      	at co.cask.hydrator.plugin.batch.sink.BatchElasticsearchSink.prepareRun(BatchElasticsearchSink.java:51) ~[na:na]
      	at co.cask.cdap.etl.batch.spark.ETLSpark.beforeSubmit(ETLSpark.java:128) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$4.call(SparkRuntimeService.java:321) ~[na:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$4.call(SparkRuntimeService.java:316) ~[na:na]
      	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:174) ~[co.cask.cdap.cdap-data-fabric-3.4.0-SNAPSHOT.jar:na]
      	... 5 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                sree Sreevatsan Raman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: