Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11970

MapReduce jobs fail to be submitted on HDP 2.6.1

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: MapReduce
    • Labels:
    • Rank:
      1|i0046f:

      Description

      MapReduce jobs that we've tested successfully against HDP 2.6.0.3 are failing (before we can even submit them) on HDP 2.6.1.0. The is due to the client jars not being localized properly for the launched container.

      Upon launching PurchaseHistoryBuilder (launched within the context of PurchaseHistoryWorkflow):

      2017-06-22 11:46:06,624 - ERROR [action-PurchaseHistoryBuilder-0:c.c.c.i.a.r.b.MapReduceRuntimeService@358] - MapReduce Program 'PurchaseHistoryBuilder' failed.
      java.lang.Exception: org/apache/hadoop/mapreduce/Job
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:358)
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:464)
              at java.lang.Thread.run(Thread.java:745)
      java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/Job
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.createJob(MapReduceRuntimeService.java:480)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:220)
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:464)
              at java.lang.Thread.run(Thread.java:745)
      java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.Job
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at co.cask.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:46)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.createJob(MapReduceRuntimeService.java:480)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:220)
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:464)
              at java.lang.Thread.run(Thread.java:745)
      

      On an HDP 2.5.3 cluster, I see the following file in the container's data directory:
      mr-framework/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-37.jar
      However, on the HDP 2.6.1 cluster, I do not see any file matching the pattern

      *hadoop-mapreduce-client-core*

      in the container's data directory. The mr-framework directory does not exist there either.

      I see this WARN in master logs when launching PurchaseHistoryWorkflow:

      2017-06-22 08:29:55,126 - WARN  [appfabric-executor-11:c.c.c.i.a.r.b.d.MapReduceContainerHelper@111] - Failed to resolve /hdp/apps/2.6.1.0/mapreduce/mapreduce.tar.gz#mr-framework URI. MapReduce framework path is not used. Check the setting for mapreduce.application.framework.path.
      java.io.FileNotFoundException: File does not exist: hdfs://10.210.0.28/hdp/apps/2.6.1.0/mapreduce/mapreduce.tar.gz
              at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:136) ~[hadoop-hdfs-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:468) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2195) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2191) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2191) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:603) ~[hadoop-common-2.7.3.2.6.1.0-129.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerHelper.getFrameworkURI(MapReduceContainerHelper.java:103) [na:na]
              at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerHelper.localizeFramework(MapReduceContainerHelper.java:126) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.DistributedMapReduceProgramRunner.setupLaunchConfig(DistributedMapReduceProgramRunner.java:96) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.DistributedWorkflowProgramRunner.setupLaunchConfig(DistributedWorkflowProgramRunner.java:133) [na:na]
              at co.cask.cdap.internal.app.runtime.distributed.DistributedProgramRunner.run(DistributedProgramRunner.java:201) [na:na]
              at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.run(AbstractProgramRuntimeService.java:122) [na:na]
              at co.cask.cdap.internal.app.services.ProgramLifecycleService.start(ProgramLifecycleService.java:326) [na:na]
              at co.cask.cdap.internal.app.services.ProgramLifecycleService.start(ProgramLifecycleService.java:299) [na:na]
              at co.cask.cdap.gateway.handlers.ProgramLifecycleHttpHandler.doPerformAction(ProgramLifecycleHttpHandler.java:372) [na:na]
              at co.cask.cdap.gateway.handlers.ProgramLifecycleHttpHandler.performAction(ProgramLifecycleHttpHandler.java:347) [na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:80) [co.cask.http.netty-http-0.16.0.jar:na]
              at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:38) [co.cask.http.netty-http-0.16.0.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [io.netty.netty-3.6.6.Final.jar:na]
              at co.cask.cdap.common.http.AuthenticationChannelHandler.messageReceived(AuthenticationChannelHandler.java:64) [na:na]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]
              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
      

      Snippet from /etc/hadoop/conf/mapred-site.xml:

      <property>
        <name>mapreduce.application.framework.path</name>
        <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
      </property>
      

      It is looking for '/hdp/apps/2.6.1.0', whereas the file in hdfs is actually named:

      hdfs dfs -ls /hdp/apps
      Found 1 items
      drwxr-xr-x   - hdfs hadoop          0 2017-06-22 08:16 /hdp/apps/2.6.1.0-129
      

      Just as reference:

      # grep -R "hdp.version" /etc/hadoop
      /etc/hadoop/conf.chef/mapred-site.xml:    <value>-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}</value>
      /etc/hadoop/conf.chef/mapred-site.xml:    <value>LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64</value>
      /etc/hadoop/conf.chef/mapred-site.xml:    <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
      /etc/hadoop/conf.chef/mapred-site.xml:    <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
      /etc/hadoop/conf.chef/mapred-site.xml:    <value>-Dhdp.version=${hdp.version}</value>
      /etc/hadoop/conf.chef/mapred-env.sh:  export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS} -Dhdp.version=2.6.1.0-129"
      /etc/hadoop/conf.chef/hadoop-env.sh:  export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS} -Dhdp.version=2.6.1.0-129"
      /etc/hadoop/conf/mapred-site.xml:    <value>-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}</value>
      /etc/hadoop/conf/mapred-site.xml:    <value>LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64</value>
      /etc/hadoop/conf/mapred-site.xml:    <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
      /etc/hadoop/conf/mapred-site.xml:    <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
      /etc/hadoop/conf/mapred-site.xml:    <value>-Dhdp.version=${hdp.version}</value>
      /etc/hadoop/conf/mapred-env.sh:  export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS} -Dhdp.version=2.6.1.0-129"
      /etc/hadoop/conf/hadoop-env.sh:  export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS} -Dhdp.version=2.6.1.0-129"
      

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: