Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10842

Hydrator should call the task context progress() to indicate status

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.5.1
    • Fix Version/s: None
    • Component/s: Pipelines
    • Labels:
    • Rank:
      1|hzzoen:

      Description

      In some scenarios, a single event from a sink can "explode" into many events to process downstream. For example, if the event is a file name that expands into all the events that are read from that named file. In this case, processing may take longer than the configured mapreduce.task.timeout.

      If a single event expands into many events, Hydrator should use the Mapper.Context(or Reducer.Context).progress() to indicate that the job is still alive.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: