Details

    • Release Notes:
      Introduces a new, event-driven scheduling system.
    • Rank:
      1|hzzo87:

      Description

      Currently, schedulers only support cron-based triggers. That means a workflow runs every so often. The disadvantage of that are

      • wasted resources in case no data is available
      • added latency in case the data is available sooner

      It would be better if a schedule could be triggered by event notifications about:

      • state change of another program (for example, workflow finished successfully)
      • data availability (for example, a partition was added to a PFS)
      • ...

      Event-based schedules may need to be combined with time schedules, for example: "run this workflow as soon as a new partition is available, but no more than 2 times an hour"

      This needs more detailed design. WIP design here: https://wiki.cask.co/pages/viewpage.action?pageId=4363504

        Attachments

          Issue Links

          1.
          Revisit app APIs Sub-task Resolved Ali Anwar
          2.
          Implement disable/enable for program schedules Sub-task Resolved Andreas Neumann
          3.
          Change job queue to use multiple columns Sub-task Resolved Ali Anwar
          4.
          Provide MessagingContext to DatasetDefinition for publishing dataset state changes Sub-task Resolved Terence Yim
          5.
          Implement Constraints for Schedules Sub-task Resolved Ali Anwar
          6.
          Write javadocs for ScheduleBuilder Sub-task Resolved Ali Anwar
          7.
          Subscribe to dataset state changes notification and run triggered jobs Sub-task Resolved Chengfeng Mao
          8.
          Add a dataset for storing and looking up schedules Sub-task Resolved Andreas Neumann
          9.
          Ensure that new schedules do not conflict with old schedules Sub-task Resolved Ali Anwar
          10.
          Port all schedule test cases to use new APIs Sub-task Resolved Chengfeng Mao
          11.
          Copy schedules in AppMetadataStore to ProgramScheduleStoreDataset Sub-task Resolved Chengfeng Mao
          12.
          Persist messageId when fetching in NotificationSubscriberService Sub-task Resolved Ali Anwar
          13.
          Refactor scheduler logic to core scheduler service Sub-task Resolved Chengfeng Mao
          14.
          TimeRangeConstraint should support a range like 10pm-6am Sub-task Resolved Ali Anwar
          15.
          Serialization of TimeZone in TimeRangeConstraint Sub-task Resolved Andreas Neumann
          16.
          Update CSD with JobQueue debug tool Sub-task Resolved Chris Gianelloni
          17.
          Reenable scheduler tests after core scheduler is fixed Sub-task Resolved Andreas Neumann
          18.
          Fix schedule deletion for deleted workflows Sub-task Resolved Chengfeng Mao
          19.
          Time notifications contain unnecessary fields Sub-task Resolved Chengfeng Mao
          20.
          Time triggers fire multiple times Sub-task Resolved Chengfeng Mao
          21.
          Scheduler tests should be more precise Sub-task Resolved Chengfeng Mao
          22.
          Remove ScheduleSpecification Sub-task Resolved Chengfeng Mao
          23.
          Ensure that the old UI (to display schedules) works with REST APIs Sub-task Resolved Ajai Narayan
          24.
          Scheduler REST API should not return 'NOT_FOUND' as a schedule status Sub-task Resolved Chengfeng Mao
          25.
          LastRunConstraint should have a way to consider only successful runs Sub-task Resolved Ali Anwar
          26.
          Implement timeout for schedules and jobs in the job queue Sub-task Resolved Ali Anwar
          27.
          Documentation for schedulers Sub-task Resolved Andreas Neumann
          28.
          Remove dependency on app spec in suspend/resume schedule Sub-task Resolved Chengfeng Mao
          29.
          LastRunConstraint uses milliseconds instead of seconds to get run record Sub-task Resolved Chengfeng Mao
          30.
          Write long-running test for scheduler Sub-task Resolved Chengfeng Mao
          31.
          Scheduler should have an API for updateSchedule() Sub-task Resolved Andreas Neumann
          32.
          ScheduleClient.list() does not return stream size schedules Sub-task Resolved Andreas Neumann
          33.
          Schedule status not preserved after migration Sub-task Resolved Chengfeng Mao

            Activity

              People

              • Assignee:
                andreas Andreas Neumann
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: