Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12905

Wrangler plugin can't emit error records to the error collector when there is a schema mismatch

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.3.2
    • Fix Version/s: None
    • Component/s: CDAP
    • Rank:
      1|i009on:

      Description

      A pipeline which processes CSV files (parse, filter, etc) into a defined structure will fail if there is a schema mismatch, rather than pass the bad records to the error collector.
      If a record has an issue (e.g. wrong column type) the pipeline is supposed to discard it by passing the record to the ErrorCollector plugin.
      I have created a dummy record in one of the file which has a string value for a column which accepts integer. When I process that file alone the pipeline works as expected, but as soon as I process multiple files the pipeline will throw and error and fail instead of managing the bad record.

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              thajdari Tony Hajdari
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: