Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5177

PartitionConsumer consumePartitions returns List with null PartitionDetails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.1
    • Fix Version/s: 3.4.0
    • Component/s: Datasets
    • Labels:
    • Release Notes:
      PartitionConsumer appropriately handles partitions that have been deleted since they were added to the working set.
    • Rank:
      1|hzz873:

      Description

      I am running into an issue where the call to consumePartitions returns a non-empty List of PartitionsDetails where each element in the list is null. Every time I call consumePartitions it returns the same list and because the list has null elements I cannot call onFinish without getting a null pointer exception.

      To reproduce the issue set the ConsumerConfiguration maxWorkingSetSize to 1000. Add two partitions to the PartitionFileSet and call consumePartitions(1) once to consume the first partition. Truncate the PartitionFileSet and call consumePartitions(1) again to retrieve the list with null PartitionDetails. Calling DropPartition on each of the partitions in the PartitionFileSet will cause the same behavior as preforming the truncate. Adding a new partition will not show up when consumePartitions(1) is called.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                jd565c Jeff Dix
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: