Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12968

Partition getting added multiple times to working set in case of append

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.3.2
    • Fix Version/s: None
    • Component/s: API, CDAP
    • Labels:
    • Rank:
      1|i00a0v:

      Description

      When a partition is getting appended, PartitionConsumer API is adding the partition again to the working set. This is particularly causing problem if a previous instance of the same partition exists in the working set from a failure. This is causing the same partition to be processed multiple times, in some cases concurrently. Here is an example of the same partition existing multiple times -
      {
      "partitionKey": {
      "fields":

      { "ET": "20171202" }

      },
      "processState": "AVAILABLE",
      "numFailures": 3,
      "timestamp": 0
      },
      {
      "partitionKey": {
      "fields":

      { "ET": "20171202" }

      },
      "processState": "AVAILABLE",
      "numFailures": 2,
      "timestamp": 0
      },
      {
      "partitionKey": {
      "fields":

      { "ET": "20171202" }

      },
      "processState": "IN_PROGRESS",
      "numFailures": 1,
      "timestamp": 1512518439916
      },
      {
      "partitionKey": {
      "fields":

      { "ET": "20171202" }

      },
      "processState": "IN_PROGRESS",
      "numFailures": 0,
      "timestamp": 1512518439916
      },

      I think before adding a ConsumablePartition to the ConsumerWorkingSet, there needs to be a check if the same partition exists already.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                srikrishna_alla Srikrishna Alla
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: