Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12758

Realtime pipeline should not fail if a Kafka broker goes down

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.3.1
    • Fix Version/s: None
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
    • Rank:
      1|i008uf:

      Description

      if a Kafka broker goes down, should the pipeline really fail? Or should it keep running or pause until that broker is back online? Also, if Kafka is replicated, it could fail over to a different broker for the partition in question. Although the answer is not obvious, it seems that pipelines should be more resilient to transient failure of a source or sink.

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: