Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14589

MetricsProcessorServiceTest is flaky

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 6.0.0
    • Fix Version/s: 6.0.0
    • Component/s: Metrics, Test
    • Labels:
    • Rank:
      1|i00j6v:

      Description

      The test starts the metrics processor service, emits a few metrics, sleeps for 500ms, then stops and restarts the service. If the service has no come up completely (not started all its threads) at the time it is stopped, this leads to the following exception: 

      testMetricsProcessor(co.cask.cdap.metrics.process.MetricsProcessorServiceTest)  Time elapsed: 3.023 sec  <<< ERROR!
      com.google.common.util.concurrent.UncheckedExecutionException: java.util.ConcurrentModificationException
      	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015)
      	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
      	at com.google.common.util.concurrent.AbstractService.stopAndWait(AbstractService.java:225)
      	at com.google.common.util.concurrent.AbstractExecutionThreadService.stopAndWait(AbstractExecutionThreadService.java:170)
      	at co.cask.cdap.metrics.process.MetricsProcessorServiceTest.testMetricsProcessor(MetricsProcessorServiceTest.java:92)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runners.Suite.runChild(Suite.java:127)
      	at org.junit.runners.Suite.runChild(Suite.java:26)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createReqestAndRun(JUnitCoreWrapper.java:139)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:111)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:84)
      	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:138)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
      	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
      	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)
      Caused by: java.util.ConcurrentModificationException
      	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
      	at java.util.ArrayList$Itr.next(ArrayList.java:851)
      	at co.cask.cdap.metrics.process.MessagingMetricsProcessorService.triggerShutdown(MessagingMetricsProcessorService.java:287)
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1.doStop(AbstractExecutionThreadService.java:76)
      	at com.google.common.util.concurrent.AbstractService.stop(AbstractService.java:198)
      	... 47 more
       

      The way this service implements startup and shutdown needs to be made thread-safe. It currently maintains a list of threads, interrupts all of them when triggering shutdown. Iterating over this list while at the same time, still starting up, adding more threads to the list, causes the concurrent modification exception.  

       

        Attachments

          Activity

            People

            • Assignee:
              yaojie Yaojie Feng
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: