Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12017

Metrics processing causes region hotspotting

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.2.0, 4.1.1
    • Fix Version/s: 4.3.0
    • Component/s: Metrics
    • Labels:
    • Release Notes:
      Added capability to salt the row key of the metrics tables so that writes are evenly distributed and there is no region hot spotting
    • Rank:
      1|i004fz:

      Description

      Metrics are written to the metrics table with a row key that is composed from all the dimensions in the metrics context and the metric name. The metric context contains the instance id of the container, which, in a MapReduce job, is the task ID of the mapper or reducer. A large MapReduce that runs 1000 mappers and emits 10 different metrics would therefore write to 10000 rows every second. Because the metrics table is not salted or otherwise spread out, all these writes (having the same prefix of namespace, app, program id, run id) go to the same region. This region goes under very heavy load, which can impact performance of other regions served on the same region server.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vinisha Vinisha Shah
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: