Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2991

Explore doesn't work when it launches map-reduce job

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|hzyvyn:

      Description

      Explore has an issue after the recent updates to twill-snapshot and dependency changes in cdap.

      2015-07-10 19:37:35,384 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
              at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
              at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
              at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
              at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
              at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
              at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
      Caused by: java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
              ... 9 more
      Caused by: java.lang.RuntimeException: Error in configuring object
              at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
              at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
              at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
              at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
              ... 14 more
      Caused by: java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
              ... 17 more
      Caused by: java.lang.RuntimeException: Map operator initialization failed
              at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
              ... 22 more
      Caused by: java.lang.NoSuchMethodError: org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.addListener(Lcom/google/common/util/concurrent/Service$Listener;Ljava/util/concurrent/Executor;)V
              at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.<init>(DefaultZKClientService.java:403)
              at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.<init>(DefaultZKClientService.java:392)
              at org.apache.twill.internal.zookeeper.DefaultZKClientService.<init>(DefaultZKClientService.java:98)
              at org.apache.twill.zookeeper.ZKClientService$Builder.build(ZKClientService.java:101)
              at co.cask.cdap.common.guice.ZKClientModule.provideZKClientService(ZKClientModule.java:53)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:104)
              at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
              at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
              at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
              at com.google.inject.Scopes$1$1.get(Scopes.java:65)
              at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
              at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
              at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
              at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
              at co.cask.cdap.hive.context.ContextManager.createContext(ContextManager.java:149)
              at co.cask.cdap.hive.context.ContextManager.getContext(ContextManager.java:85)
              at co.cask.cdap.hive.stream.StreamSerDe.initialize(StreamSerDe.java:88)
              at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:307)
              at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:353)
              at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:123)
              ... 22 more
      

      In org.apache.twill.internal.zookeeper.DefaultZKClientService, we use com.google.common.util.concurrent.AbstractService#addListener, which isn't present in older guava versions.

      the explore-query container's classpath has job.jar before the jar files in the container directory, and since job.jar has guava-11, we use that and get NoSuchMethodError exception.

      however, we did not update guava version from twill-0.5.0 to twill-0.6.0-SNAPSHOT, its just the new changes started exposing the issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shankar Shankar Selvam
                Reporter:
                shankar Shankar Selvam
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: