Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7612

Spark programs cannot be stopped after master failover/restart

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0, 3.6.1, 3.5.2
    • Component/s: App Fabric
    • Labels:
      None
    • Release Notes:
      Fixes cases where Spark programs cannot be started after a master failover or restart.
    • Rank:
      1|hzzodr:

      Description

      To reproduce:

      Start a spark program (realtime Hydrator pipeline works)

      Restart cdap-master

      Stop the spark program

      The CDAP program status will be stopped, but the yarn containers will still be running. I believe the cause is that DistributedProgramRuntimeService does not create a controller for spark programs discovered from the TwillRunner.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: