Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6545

Log Saver slows down when Kafka partitions are highly skewed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.3, 3.3.5
    • Fix Version/s: 3.5.0, 3.4.4, 3.3.6
    • Component/s: Log
    • Labels:
      None
    • Rank:
      1|hzzgg7:

      Description

      We have seen cases when amount of logging in apps is widely varied, some Kafka partitions can become really huge (800x) compared to others. Since Log Saver uses a single data structure to combine logs from all partition, the large partition will affect other partitions too.

      We should use separate data structure to store logs of each partition so that the processing of each partition can happen in parallel.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: