Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14372

Dataprep pipelines should not require directives

    XMLWordPrintableJSON

    Details

    • Release Notes:
      Fixed dataprep transform to make directives optional. If none are given, the transform is a no-op.
    • Rank:
      1|i00hwn:

      Description

      Scenario: Cloud migration from BigQuery to S3

      User wants to backup data in bigquery into S3, to do this they start from browsing bigquery table in dataprep and create pipelines from that and attach an S3 sink 

       

      This does not work out of box because a wrangler sink gets attached even if the user has not done any transformation. 

       

      Potential fixes:

      1. Allow empty recipes 
      2. If there are no directives in data prep stage do not add a wrangler transform step while going from data prep to data pipelines

       

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: