Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14558

Realtime/Spark Streaming pipelines ignore config and always run with checkpointing on

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.1.0, 5.0.0, 4.3.3, 4.3.4
    • Fix Version/s: 4.3.5, 5.1.2
    • Component/s: Pipelines, Spark
    • Labels:
      None
    • Release Notes:
      Fixed a bug that pipeline checkpointing is always on regardless of the option for the realtime pipeline.
    • Rank:
      1|i00j0n:

      Description

      In the Pipeline UI when running a Realtime Pipeline you have an option to turn on/off checkpointing in Spark Streaming. However, regardless of this configuration, the pipeline runs with checkpointing on.

      Upon inspection of code, it looks like this is hard-coded in the SparkStreamingPipelineDriver:

      https://github.com/caskdata/cdap/blob/develop/cdap-app-templates/cdap-etl/cdap-data-streams/src/main/java/co/cask/cdap/datastreams/SparkStreamingPipelineDriver.java#L129

      SparkStreamingPipelineRunner runner = new SparkStreamingPipelineRunner(sec, jssc, pipelineSpec, false);
      

      Which has the final argument (shown below as checkpointingDisabled) hard-coded to false:

      public SparkStreamingPipelineRunner(JavaSparkExecutionContext sec, JavaStreamingContext streamingContext, DataStreamsPipelineSpec spec, boolean checkpointsDisabled)
      

      An additional impact to this is that macros do not function whatsoever in realtime pipelines because they are disabled in realtime pipelines when checkpointing is turned on, and there is no way to turn off checkpointing, thus always preventing macros from working.

        Attachments

          Activity

            People

            • Assignee:
              jon Jon Gray
              Reporter:
              jon Jon Gray
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: