Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11848

Pipelines with a sparkprogram and sparkcompute may fail to deploy

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: Pipelines
    • Labels:
      None
    • Rank:
      1|i003h3:

      Description

      If a pipeline uses a sparkprogram plugin and a sparkcompute plugin, it may fail to deploy with exception:

      Caused by: java.lang.NullPointerException: null
      	at co.cask.cdap.datapipeline.ExternalSparkProgram.configure(ExternalSparkProgram.java:55) ~[na:na]
      	at co.cask.cdap.api.spark.AbstractSpark.configure(AbstractSpark.java:45) ~[na:na]
      	at co.cask.cdap.app.DefaultAppConfigurer.addSpark(DefaultAppConfigurer.java:170) ~[na:na]
      	at co.cask.cdap.datapipeline.SmartWorkflow.addProgram(SmartWorkflow.java:328) ~[na:na]
      	at co.cask.cdap.datapipeline.SmartWorkflow.addPrograms(SmartWorkflow.java:241) ~[na:na]
      	at co.cask.cdap.datapipeline.SmartWorkflow.configure(SmartWorkflow.java:177) ~[na:na]
      	at co.cask.cdap.api.workflow.AbstractWorkflow.configure(AbstractWorkflow.java:58) ~[na:na]
      	at co.cask.cdap.app.DefaultAppConfigurer.addWorkflow(DefaultAppConfigurer.java:182) ~[na:na]
      	at co.cask.cdap.api.app.AbstractApplication.addWorkflow(AbstractApplication.java:128) ~[na:na]
      	at co.cask.cdap.datapipeline.DataPipelineApp.configure(DataPipelineApp.java:76) ~[na:na]
      	at co.cask.cdap.api.app.AbstractApplication.configure(AbstractApplication.java:63) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.getSpecJson(InMemoryConfigurator.java:162) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.createResponse(InMemoryConfigurator.java:128) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.config(InMemoryConfigurator.java:118) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:104) ~[na:na]
      	... 20 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: