Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0
    • Component/s: Pipelines
    • Labels:
      None
    • Rank:
      1|hzz89b:

      Description

      We need a planner that will take a logical pipeline and break it down into a physical workflow.

      The contract will be to take the ETLConfig object, then turn it into one or more Pipeline objects that the individual programs will understand and know how to implement. So ETLMapReduce will take a single Pipeline in its constructor instead of the ETLBatchConfig object it used to take. ETLMapReduce only needs to know how to execute a single Pipeline.

      Terminology is confusing now, should rename the Pipeline object to PipelinePhase or something like that.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: