Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3101

Workflow runs are not getting scheduled due Quartz exceptions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0
    • Component/s: CDAP
    • Labels:
      None
    • Release Notes:
      Increased the default value of the quartz thread pool size to 100.
    • Rank:
      1|hzywkn:

      Description

      Seeing the below exception in master logs -

      2015-07-21 12:10:00,004 - ERROR [SimpleQuartzScheduler_QuartzSchedulerThread:o.q.c.QuartzSchedulerThread@441] - Runtime error occurred in main trigger firing loop.
      java.util.concurrent.RejectedExecutionException: Task org.quartz.core.JobRunShell@7a960e6d rejected from java.util.concurrent.ThreadPoolExecutor@118d54fb[Running, pool size = 30, active threads = 30, queued tasks = 0, completed tasks = 54]
              at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) ~[na:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) ~[na:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) ~[na:1.7.0_75]
              at co.cask.cdap.internal.app.runtime.schedule.ExecutorThreadPool.runInThread(ExecutorThreadPool.java:44) ~[co.cask.cdap.cdap-
      app-fabric-3.2.0-SNAPSHOT.jar:na]
              at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394) ~[org.quartz-scheduler.quartz-2.2.0.jar:na]
      

      Also, noticed that multiple instances of some workflows are running for about a week -
      /v3/namespaces/default/apps/Wise/workflows/WiseWorkflow/runs?status=RUNNING

      [
          {
              "properties": {}, 
              "runid": "26f075e1-3053-11e5-88a0-c81f66e2678c", 
              "start": 1437556939, 
              "status": "RUNNING", 
              "twillrunid": "82565f90-01b4-4284-a7d5-05788153b586"
          }, 
          {
              "properties": {}, 
              "runid": "39b7ef9c-3054-11e5-a4e2-c81f66e2678c", 
              "start": 1437557400, 
              "status": "RUNNING", 
              "twillrunid": "dbe29b27-f32b-43f3-954d-3dcba4534b8c"
          }, 
          {
              "properties": {}, 
              "runid": "6ae8b8f2-3058-11e5-8c7c-c81f66e2678c", 
              "start": 1437559201, 
              "status": "RUNNING", 
              "twillrunid": "15e2a2f9-88a6-4085-8d51-974110698f44"
          }, 
          {
              "properties": {}, 
              "runid": "9f3d0d58-3055-11e5-9148-c81f66e2678c", 
              "start": 1437558000, 
              "status": "RUNNING", 
              "twillrunid": "b00a1cac-1b78-4e1f-93e3-5ed8ff5acacc"
          }, 
          {
              "properties": {}, 
              "runid": "cb020240-32f9-11e5-9d5d-c81f66e2678c", 
              "start": 1437848413, 
              "status": "RUNNING", 
              "twillrunid": "7c7e919e-b7b4-4ccd-bc31-768611a64eb2"
          }
      ]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sagar Sagar Kapare
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: