Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13643

Supporting field lineage for the streaming pipelines.

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.0
    • Fix Version/s: 6.2.0, 6.1.2
    • Component/s: CDAP, Pipelines
    • Labels:
      None
    • Release Notes:
      Added the feature to record field lineage for streaming pipelines.
    • Rank:
      1|i00dz3:

      Description

      Need to figure out way to record the field lineage operations for the streaming pipelines. There are few issues in doing so currently - 

      1. Lifecycle for the streaming pipelines is different than batch, for example we record the lineage once the batch program is complete, however there is no completion of the streaming pipelines
      2. Because of checkpointing, once started, runtime arguments of the streaming pipelines cannot be changed, so each run will record the same set of operations. We may not want to record operations per run then.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yaojie Yaojie Feng
                Reporter:
                sagar Sagar Kapare
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: