Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12074

Support repartitioning Kafka or switching to a new Kafka topic

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.2.0
    • Fix Version/s: None
    • Component/s: Kafka, Log
    • Rank:
      1|i004rb:

      Description

      A Kafka topic, once created, does not scale elastically by adding more brokers. When the workload grows, more brokers may be needed. One way to do this would be to delete the current Kafka topic (logs-user.v2), bring up more brokers and restart CDAP master, which will then recreate the topic, and ll new brokers can participate. However, that would discard all in-flight logs. Instead, CDAP should have an API to create a new topic, and use that for future logs, while still processing the logs from the old topic until it is drained.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bhooshan Bhooshan Mogal
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: