Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3092

Read multiple files with one mapper in FileBatchSource

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: CDAP
    • Labels:
      None
    • Rank:
      1|hzywiv:

      Description

      When reading in a large number of small files, FileBatchSource will use one mapper for each file. This leads to unreasonably slow runtime and out of memory issues. Fix so that one mapper reads many files.

        Attachments

          Activity

            People

            • Assignee:
              jay Jay Jin
              Reporter:
              jay Jay Jin
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: