Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11959

Add a configuration for the frequency of getting MapReduce task reports

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.2.0, 4.1.1
    • Fix Version/s: 4.2.1, 4.1.2
    • Component/s: MapReduce
    • Labels:
    • Release Notes:
      Adds a way to limit the frequency of retrieving the MapReduce task report, which could cause network load for very large jobs.
    • Rank:
      1|i0044n:

      Description

      The MapReduceRuntimeService constantly polls the MapReduce job for completion, then gets the task report to emit metrics, then sleeps for a second. If a job has 10s of thousands of tasks, the task report becomes hundreds of MB large, which can cause OutOfMemory, it also creates high load on the network.

      The frequency of getting the task report should be configurable to reduce this load.

        Attachments

          Activity

            People

            • Assignee:
              andreas Andreas Neumann
              Reporter:
              andreas Andreas Neumann
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: