Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12663

Deleting Metrics can incur HBase scanner timeout

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.3.0
    • Fix Version/s: None
    • Component/s: Metrics
    • Labels:
    • Rank:
      1|i0089z:

      Description

      here is the symptom in the logs:

      java.lang.RuntimeException: org.apache.hadoop.hbase.client.ScannerTimeoutException: 1095075ms passed since the last invocation, timeout is currently set to 60000
              at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.HBaseScanner.next(HBaseScanner.java:84) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.CombinedMetricsScanner.<init>(CombinedMetricsScanner.java:55) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.CombinedHBaseMetricsTable.scan(CombinedHBaseMetricsTable.java:185) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.timeseries.FactTable.getScanner(FactTable.java:214) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.timeseries.FactTable.delete(FactTable.java:222) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.cube.DefaultCube.delete(DefaultCube.java:238) ~[na:na]
              at co.cask.cdap.metrics.store.DefaultMetricStore.delete(DefaultMetricStore.java:334) ~[na:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deleteMetrics(ApplicationLifecycleService.java:626) ~[na:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deleteApp(ApplicationLifecycleService.java:709) ~[na:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.removeAppInternal(ApplicationLifecycleService.java:554) ~[na:na]
              at co.cask.cdap.internal.app.services.ApplicationLifecycleService.removeApplication(ApplicationLifecycleService.java:542) ~[na:na]
              at co.cask.cdap.gateway.handlers.AppLifecycleHttpHandler.deleteApp(AppLifecycleHttpHandler.java:297) ~[na:na]
              at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) ~[na:na]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77]
              at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:80) ~[co.cask.http.netty-http-0.16.0.jar:na]
              at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:38) [co.cask.http.netty-http-0.16.0.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [io.netty.netty-3.6.6.Final.jar:na]
              at co.cask.cdap.common.http.AuthenticationChannelHandler.messageReceived(AuthenticationChannelHandler.java:64) [na:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
      

      The cause is that by default, the HBase client caches 1000 rows, by the time a next() call hits the HBase server again, HBase has timed out the scan.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vinisha Vinisha Shah
                Reporter:
                andreas Andreas Neumann
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: