Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6215

ConcurrentPartitionConsumer WorkingSet does not aged out partitions that no longer exists

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5.0, 3.3.4
    • Fix Version/s: 3.5.1
    • Component/s: Workflow
    • Labels:
    • Release Notes:
      PartitionConsumer appropriately drops partitions that have been deleted from a corresponding PartitionedFileSet
    • Rank:
      1|hzzehz:

      Description

      Partitions that are in the WorkingSet and are later dropped will remain in the WorkingSet forever. To reproduce, added two partitions to a PartitionedFileSet and set the ConcurrentPartitionConsumer's ConsumerConfiguration with a maxWorkingSetSize of 1 and the timeout of 60. Execute consumePartitions and mark the partition as failed with onFinish. Drop the partition that was marked as failed. Call consumePartitions again and nothing will return because the failed partition that was dropped is still in the workingSet.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                jd565c Jeff Dix
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: