Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14724

Action only pipeline fails to run with dataproc profile

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.0
    • Component/s: Cloud Provisioner
    • Labels:
      None
    • Release Notes:
      Fixed a bug that caused action only pipelines to fail when running using a cloud profile.
    • Rank:
      1|i00jzr:

      Description

      When I try to run a pipeline that contains a single GCS Copy action, the pipeline consistently fails to start with nothing in the program logs. If I set a breakpoint in CDAP, I can see that the remote process fails to start with:

      2019-01-08 02:19:01,519 - WARN  [main:c.c.c.c.c.Configuration@1814] - file:/home/cdap/6c4d4a98-12eb-11e9-b025-02427834a638/cConf.xml:an attempt to override final parameter: messaging.system.topics;  Ignoring.
      2019-01-08 02:19:01,521 - WARN  [main:c.c.c.c.c.Configuration@1814] - file:/home/cdap/6c4d4a98-12eb-11e9-b025-02427834a638/cConf.xml:an attempt to override final parameter: app.program.runtime.monitor.topics.configs;  I
      gnoring.
      2019-01-08 02:19:01,521 - WARN  [main:c.c.c.c.c.Configuration@1814] - file:/home/cdap/6c4d4a98-12eb-11e9-b025-02427834a638/cConf.xml:an attempt to override final parameter: messaging.max.instances;  Ignoring.
      2019-01-08 02:19:01,522 - WARN  [main:c.c.c.c.c.Configuration@1814] - file:/home/cdap/6c4d4a98-12eb-11e9-b025-02427834a638/cConf.xml:an attempt to override final parameter: stream.instance.file.prefix;  Ignoring.
      2019-01-08 02:19:01,522 - WARN  [main:c.c.c.c.c.Configuration@1814] - file:/home/cdap/6c4d4a98-12eb-11e9-b025-02427834a638/cConf.xml:an attempt to override final parameter: app.program.runtime.monitor.server.info.file; 
       Ignoring.
      2019-01-08 02:19:05,386 - ERROR [main:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@203] - org/apache/zookeeper/Watcher
      java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher
              at org.apache.twill.zookeeper.ZKClientService$Builder.build(ZKClientService.java:101) ~[org.apache.twill.twill-zookeeper-0.13.0.jar:0.13.0]
              at co.cask.cdap.common.guice.ZKClientModule.provideZKClientService(ZKClientModule.java:59) ~[na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
              at com.google.inject.internal.ProviderMethod$ReflectionProviderMethod.doProvision(ProviderMethod.java:299) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:172) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) ~[com.google.inject.guice-4.0.jar:na]
              at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051) ~[com.google.inject.guice-4.0.jar:na]
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.createCoreServices(AbstractProgramTwillRunnable.java:504) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.doInitialize(AbstractProgramTwillRunnable.java:255) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.initialize(AbstractProgramTwillRunnable.java:200) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.doMain(AbstractProgramTwillRunnable.java:167) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.WorkflowTwillRunnable.main(WorkflowTwillRunnable.java:55) [na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
              at co.cask.cdap.internal.app.runtime.distributed.remote.RemoteLauncher.main(RemoteLauncher.java:73) [launcher.jar:na]
      Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher
              at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_181]
              at co.cask.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:44) ~[na:na]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_181]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_181]
              ... 31 common frames omitted
      

      A pipeline that has the same action followed by a GCS source to GCS sink does not run into the same issue. I'm not sure if it's some type of race condition (maybe pipelines with mapreduce/spark take longer to setup and by then the required zk libraries are accessible?) or if this has something to do with the differing sets of dependencies that get shipped to the remote cluster if there is a mapreduce/spark program in the pipeline.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: