Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11218

Hydrator should not keep transformed records in memory

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 4.0.0
    • Component/s: Pipelines
    • Labels:
    • Rank:
      1|hzzjt3:

      Description

      Hydrator app uses TransformDetail to store all the transformed records in memory for a pipeline. As a result, pipeline will need more container memory. For example, join with a couple of GBs of data requires more container memory. Attached is the sample pipeline.

        Attachments

          Activity

            People

            • Assignee:
              vinisha Vinisha Shah
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: