Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2747

PartitionedFileSet should index the creation time of partitions to allow selection of partitions that were created after a given time

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: Datasets
    • Labels:
      None
    • Rank:
      1|hzyujb:

      Description

      This will allow applications (or schedulers) to check for new data periodically. Instead of a full table scan of all partitions, it can then remember the latest partition it has seen, and next time only look for partitions that were created after that.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: