Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7731

Add fetch size and transaction flush interval configuration for Kafka Consumer Flowlet

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5.2
    • Fix Version/s: 4.2.0, 4.1.1, 3.5.4, 4.0.2
    • Component/s: CDAP Services
    • Labels:
    • Release Notes:
      Added fetch size and transaction flush interval configurations to the Kafka Consumer Flowlet.
    • Rank:
      1|hzzpg7:

      Description

      When a KafkaConsumerflowlet starts after failover, and when there are huge number of messages left in the topic to be picked up, the flowlet tries to process all the backlog of messages at once and because of their existing limitation on Transaction timeout, we get in to a infinite loop of “Transaction timeout Exceptions”

      We need at the minimum two configurable properties in Kafka Consumer Flowlet

      1. Able to configure number of messages to fetch in a single batch
      2. Able to configure a time interval for periodic transaction commits

        Attachments

          Activity

            People

            • Assignee:
              sagar Sagar Kapare
              Reporter:
              0012393 Brett Lamport
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: