Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11533

Reduce the timeout of Explore queries

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: Explore
    • Labels:
      None
    • Rank:
      1|i001m7:

      Description

      When a user starts an Explore query, the results are stored until the user downloads the query result. We have a timeout, explore.active.operation.timeout.secs, after which we clean up the query results to prevent the query results from staying forever.

      The timeout used today is 24 hours, which is equal to the timeout of long transaction. This leads to a race sometimes when the transaction might timeout before the query is cleanup is done. In such cases we see the below exception. It would be good to reduce the cleanup duration to 23 hours to prevent the long transaction from timing out.

      2017-05-17 01:25:30,644 - ERROR [explore-handle-timeout:o.a.t.d.TransactionServiceThriftClient@154] - Got exception while closing transaction.
      org.apache.tephra.TransactionNotInProgressException: canCommit() is called for transaction 1494897925283000000 that is not in progress (it is known to be invalid)
              at org.apache.tephra.distributed.TransactionServiceThriftClient.commit(TransactionServiceThriftClient.java:154)
              at org.apache.tephra.distributed.TransactionServiceClient$5.execute(TransactionServiceClient.java:318)
              at org.apache.tephra.distributed.TransactionServiceClient$5.execute(TransactionServiceClient.java:314)
              at org.apache.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:219)
              at org.apache.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:188)
              at org.apache.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:313)
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.closeTransaction(BaseHiveExploreService.java:1532)
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.cleanUp(BaseHiveExploreService.java:1505)
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.closeInternal(BaseHiveExploreService.java:1211)
              at co.cask.cdap.explore.service.hive.ActiveOperationRemovalHandler$ResourceCleanup$1.call(ActiveOperationRemovalHandler.java:84)
              at co.cask.cdap.explore.service.hive.ActiveOperationRemovalHandler$ResourceCleanup$1.call(ActiveOperationRemovalHandler.java:62)
              at co.cask.cdap.security.impersonation.ImpersonationUtils$1.run(ImpersonationUtils.java:46)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
              at co.cask.cdap.security.impersonation.ImpersonationUtils.doAs(ImpersonationUtils.java:43)
              at co.cask.cdap.explore.service.hive.ActiveOperationRemovalHandler$ResourceCleanup.run(ActiveOperationRemovalHandler.java:62)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: