Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12084

Ability for DynamicPartitioner to append to an existing PFS partition

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.0
    • Component/s: Datasets, MapReduce
    • Labels:
    • Release Notes:
      Allow appending to (or overwriting) a PartitionedFileSet's partitions when using DynamicPartitioner APIs. Introduced a PartitionedFileSet.setMetadata API which now allows modifying partitions' metadata.
    • Rank:
      1|i004tz:

      Description

      This would be useful if new data arrives for an existing partition key after the partition has already been created.

      It is not clear what this would mean for partition consumer and the "create" time stamp - do we need to add another field "last modified"?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: