Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-15127

Programs stop getting scheduled after a few hours

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.0.0
    • Fix Version/s: 6.0.0
    • Component/s: App Fabric, Scheduler
    • Labels:
      None
    • Release Notes:
      Fixed a race condition in the remote runtime scp implementation that can causing process hanging
    • Rank:
      1|i00mh3:

      Description

      I have 25 pipelines scheduled to run every minute with a max concurrent runs constraint of 1. The pipelines get scheduled correctly for about 12 hours. After that the pipelines gets stuck in provisioning state, and the no more runs gets scheduled.

      I have attached the stack trace and the logs of appfabric server (from 2019-03-24 06:20 to 2019-03-24 06:39), also the program log of the program that got stuck in provisioning state. The CDAP instance was started at 2019-03-23 18:22, the schedules stopped getting scheduled around 2019-03-24 06:30.

       

        Attachments

        1. appfabric.jstack
          181 kB
        2. app-fabric.log
          517 kB
        3. program.log
          9 kB

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: