Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-8094

Limiting number of records will limit empty RDD for data streaming app for preview

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0
    • Fix Version/s: 4.1.0, 4.0.1
    • Component/s: None
    • Labels:
      None
    • Release Notes:
      Fixed an issue where the number of records processed during a preview run of the realtime data pipeline was being incremented incorrectly.
    • Rank:
      1|hzzsdb:

      Description

      In preview mode, if we run a data streaming realtime pipeline, we are not limiting the number of records correctly. For each batch interval, we increment the number of records even if the RDD is empty, which will result in much fewer results in PreviewStore.
      Now we will change the behavior to limit the number of records for each RDD similar to Mapreduce we limit the number of records per mapper.

        Attachments

          Activity

            People

            • Assignee:
              yaojie Yaojie Feng
              Reporter:
              yaojie Yaojie Feng
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: