Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5536

ConsumablePartition should have a way to "untake" a partition

    XMLWordPrintableJSON

    Details

    • Release Notes:
      Allow user to put back partitions to PartitionConsumer, without processing it.
    • Rank:
      1|hzy5uf:

      Description

      Right now the only way to put a partition back into the working set after it has been taken is to call retry(). That, however, bumps the failure count of the partition by one, which may not be desired. We need a way to put it back without increasing the failures.

      The use case for this is when the PartitionAcceptor cannot decide what partitions to take; for example if you consume two PFS as multiple inputs to a map/reduce job that joins them. To align the logical times of both inputs, you would need to first see what partitions are available in each of the two file sets, then use only those for which the corresponding partitions are available in both file sets. The remaining partitions need to put back into the working set until they are ready to be processed.

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: