Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10959

FileBatchSource should support recursive reading of directories

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 4.1.0
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
    • Sprint:
      App Eng Sprint 4, App Eng Sprint 5, App Eng Sprint 6
    • Release Notes:
      Added the ability to read recursive directories to the File source plugin.
    • Rank:
      1|hzzsnz:

      Description

      Currently, this can be set with the file system properties by setting them to

       { "mapreduce.input.fileinputformat.input.dir.recursive": "true" }
      

      However, that does not work in conjunction with a file regex.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ananya Ananya Bhattacharya
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: