Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14528

Improvements to processed records metrics for each stage in a pipeline



    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 6.1.0
    • Component/s: Pipelines
    • Labels:
    • Rank:


      When records are processed by a stage in a pipeline, different metrics are emitted such as:

      • Rate - num of records processed per second
        • Its possible that¬†this number is non integer. We should round it up because it shows number of records processed.
        • This value provides rate of transform method for a given stage. For source and sink where Inputformat/Outputformat is responsible for reading/writing this value to actual sink, this value will be pretty high. However, it will not provide insights on how much time it actually took to write to actual sink. We should rethink how(if at all), we should show these metrics for sources and sinks.
        • Also if backend has more timeseries values for these metrics graph is shown on UI otherwise only numbers are shown. Its confusing user experience as user will not know why graph is not being shown.
      • Min process time
        • Its possible that min process time is 0 (as shown in the screenshot). It should be shown in milli seconds because it will not be zero if the transform method processed the records.
      • Max process time
      • ....¬†




            • Assignee:
              poorna Poorna Chandra
              vinisha Vinisha Shah
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: