Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-16823

Handle case in DataprocRuntimeJobManager where the system bucket is not present

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 6.2.1, 6.3.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|i00xcv:

      Description

      Handle case in DataprocRuntimeJobManager where the system bucket is not present. We may probably catch gcs not found exception and provide better error message. Current implementation shows this in logs where its not very intuitive from error message that the bucket is not present.

      2020-05-15 02:34:13,682 - WARN  [runtime-scheduler-7:i.c.c.r.s.c.DataprocUtils@66] - GCS path cdap-job/2bf9779d-9654-11ea-90c0-5a195f30eeeb was not cleaned up for bucket cdf-launcher due to Not Found. 
      com.google.cloud.storage.StorageException: Not Found
      	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:370) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:376) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:373) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[com.google.api.gax-1.51.0.jar:1.51.0]
      	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[com.google.cloud.google-cloud-core-1.91.3.jar:1.91.3]
      	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[com.google.cloud.google-cloud-core-1.91.3.jar:1.91.3]
      	at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:372) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:328) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	at io.cdap.cdap.runtime.spi.common.DataprocUtils.deleteGCSPath(DataprocUtils.java:53) ~[io.cdap.cdap.cdap-runtime-ext-dataproc-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.runtime.spi.runtimejob.DataprocRuntimeJobManager.launch(DataprocRuntimeJobManager.java:196) [io.cdap.cdap.cdap-runtime-ext-dataproc-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.RuntimeJobTwillPreparer.launch(RuntimeJobTwillPreparer.java:86) [na:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.AbstractRuntimeTwillPreparer.lambda$start$1(AbstractRuntimeTwillPreparer.java:466) [na:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService$ControllerFactory.lambda$create$0(RemoteExecutionTwillRunnerService.java:504) ~[na:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_252]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_252]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_252]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_252]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_252]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_252]
      	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_252]
      Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
      {
        "code" : 404,
        "errors" : [ {
          "domain" : "global",
          "message" : "Not Found",
          "reason" : "notFound"
        } ],
        "message" : "Not Found"
      }
      	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1092) ~[com.google.http-client.google-http-client-1.33.0.jar:1.33.0]
      	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[com.google.api-client.google-api-client-1.25.0.jar:1.25.0]
      	at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:360) ~[com.google.cloud.google-cloud-storage-1.101.0.jar:1.101.0]
      	... 19 common frames omitted
      2020-05-15 02:34:13,715 - ERROR [runtime-scheduler-7:i.c.c.i.a.r.d.r.RemoteExecutionTwillRunnerService@528] - Fail to start program run program_run:default.sd_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.2bf9779d-9654-11ea-90c0-5a195f30eeeb
      java.lang.Exception: Error while launching job default_sd_v2_DataPipelineWorkflow_2bf9779d-9654-11ea-90c0-5a195f30eeeb on cluster cdap-sdv2-2bf9779d-9654-11ea-90c0-5a195f30eeeb
      	at io.cdap.cdap.runtime.spi.runtimejob.DataprocRuntimeJobManager.launch(DataprocRuntimeJobManager.java:197) ~[na:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.RuntimeJobTwillPreparer.launch(RuntimeJobTwillPreparer.java:86) ~[na:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.AbstractRuntimeTwillPreparer.lambda$start$1(AbstractRuntimeTwillPreparer.java:466) ~[na:na]
      	at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService$ControllerFactory.lambda$create$0(RemoteExecutionTwillRunnerService.java:504) ~[na:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_252]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_252]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_252]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_252]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_252]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_252]
      	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_252] 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vinisha Vinisha Shah
                Reporter:
                vinisha Vinisha Shah
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: