Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12671

Transaction state cache should wait for refresh thread to stop

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.1
    • Fix Version/s: 4.3.1
    • Component/s: Datasets
    • Labels:
      None
    • Rank:
      1|i008br:

      Description

      CDAP-12454 changed the way configuration table is obtained for reading in CoprocessorCConfigurationReader. It now gets the table from CoprocessorEnvironment.

      HBase adds table that are obtained using CoprocessorEnvironment into an open tables list. During shutdown, the open tables from the list are closed.

      Transaction state cache refresh thread uses CoprocessorCConfigurationReader to read the configuration. When the state cache is stopped, the refresh thread is interrupted. However, there is no wait for the refresh thread to stop. This can lead to a race condition where the refresh thread tries to get the configuration table when the region is closing, thus modifying open tables list, which can lead to concurrent modification exception on open tables iterator.

      Exception stack trace:

      java.util.ConcurrentModificationException
      	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
      	at java.util.ArrayList$Itr.next(ArrayList.java:831)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.shutdown(CoprocessorHost.java:442)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:154)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:272)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$5.postEnvCall(RegionCoprocessorHost.java:494)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1759)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:487)
      	at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1482)
      	at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1336)
      	at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1287)
      	at co.cask.cdap.data2.increment.hbase11.IncrementSummingScannerTest.testWithBatchLimit(IncrementSummingScannerTest.java:373)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: