Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13777

Ability to archive the old field lineage records from the FieldLineageDataset.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.0.0
    • Fix Version/s: None
    • Component/s: CDAP
    • Labels:
      None
    • Rank:
      1|i00em7:

      Description

      We keep adding the records to the FieldLineageDataset and serve the information in it from REST apis. However over the period of time the data in the dataset would grow large, causing slowness in the REST api response. One approach to fix this is we can archive old data, so that dataset contains only certain (configurable) amount of historical data. Lineage records older than this data can be fetched by batch job. 

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              sagar Sagar Kapare
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: