Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10998

File Source should be able to ignore absent files/folders

    Details

    • Sprint:
      App Eng Sprint 2, App Eng Sprint 3
    • Release Notes:
      File Batch source in Hydrator now ignores empty directories
    • Rank:
      1|hzy40n:

      Description

      If there are multiple file sources in a pipeline, with each source having a different frequency of producing new data, then on any given run some sources may not have new files/folders for reading. In that case file source should have a way to ignore the not present files and continue the processing with whatever data that is available.

      Today the source that does not have new data throws exception saying it is not able to find new file/folder.

        Attachments

          Activity

            People

            • Assignee:
              ananya Ananya Bhattacharya
              Reporter:
              poorna Poorna Chandra
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: