Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2757

MapReduce should be able to write to multiple partitions of a file set

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: App Fabric, Datasets
    • Labels:
    • Release Notes:
      Ability to dynamically write to multiple partitions of a PartitionedFileSet dataset, as the output of a MapReduce job.
    • Rank:
      1|hzyukv:

      Description

      For example, if one of the dimensions of the partitioned file set is Country, but the input contains data from many countries. The output should then be written to separate files for each countries, and each file is registered as a partition at the end of the job. Hive/HCatalog calls this dynamic partitioning.

      Note that the values for the dynamic dimensions are not known ahead of time.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: