Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11027

Add offset and partition settings to Kafka streaming source

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
      None
    • Release Notes:
      Enhanced the Kafka streaming source to support configurable partitions and initial offsets, and to support optionally including the partition and offset in the output records.
    • Rank:
      1|hzzmun:

      Description

      Once checkpointing is implemented in the app, it will make sense to add offset properties to the Kafka streaming source, to allow users to start from an arbitrary offset instead of always using the latest offset.

      The current plugin is also missing properties that let users choose which partitions of a topic to read from. It should allow users to specify which topics, and to optionally specify an offset per partition.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: