Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7663

CDAP programs fails to run on EMR 4.6.0.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0, 3.6.1
    • Fix Version/s: 4.0.0, 3.6.1
    • Component/s: Distribution
    • Labels:
      None
    • Release Notes:
      Add support for Amazon EMR 4.6.0.
    • Rank:
      1|hzzou7:

      Description

      CDAP programs fails to run on EMR 4.6.0. I tested Service program type. The yarn application shows up as failed.

      stdout:

      UnJar appMaster.jar to tmp/twill.launcher-1479420980044-0
      Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain) with classpath: [file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/classes, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/resources, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/ch.qos.logback.logback-classic-1.0.9.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/ch.qos.logback.logback-core-1.0.9.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/co.cask.cdap.cdap-api-4.0.0-SNAPSHOT.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/co.cask.cdap.cdap-api-common-4.0.0-SNAPSHOT.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/co.cask.cdap.cdap-app-fabric-4.0.0-SNAPSHOT.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/co.cask.cdap.cdap-common-4.0.0-SNAPSHOT.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/com.101tec.zkclient-0.3.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/com.google.code.findbugs.jsr305-2.0.1.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/com.google.code.gson.gson-2.2.4.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/com.google.guava.guava-13.0.1.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/com.yammer.metrics.metrics-core-2.2.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/io.netty.netty-3.6.6.Final.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/javax.ws.rs.javax.ws.rs-api-2.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/log4j.apache-log4j-extras-1.2.17.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/log4j.log4j-1.2.17.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/net.jpountz.lz4.lz4-1.3.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/net.sf.jopt-simple.jopt-simple-3.2.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.kafka.kafka-clients-0.8.2.2.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.kafka.kafka_2.10-0.8.2.2.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.tephra.tephra-api-0.10.0-incubating-SNAPSHOT.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-api-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-common-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-core-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-discovery-api-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-discovery-core-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-yarn-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.apache.twill.twill-zookeeper-0.8.0.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.codehaus.janino.commons-compiler-2.7.6.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.codehaus.janino.janino-2.7.6.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.ow2.asm.asm-all-5.0.3.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.scala-lang.scala-library-2.10.4.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.slf4j.slf4j-api-1.7.5.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/org.xerial.snappy.snappy-java-1.1.1.7.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/tmp/twill.launcher-1479420980044-0/lib/zookeeper-3.4.8.jar, file:/mnt1/yarn/usercache/cdap/appcache/application_1479407581488_0045/container_1479407581488_0045_01_000001/]
      Cleanup directory tmp/twill.launcher-1479420980044-0
      

      stderr (this is all it had):

      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
              at java.lang.Class.getDeclaredMethods0(Native Method)
              at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
              at java.lang.Class.getMethod0(Class.java:2866)
              at java.lang.Class.getMethod(Class.java:1676)
              at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:86)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              ... 5 more
      

      Contents of launcher.jar::application-classpath:

      
              $HADOOP_CONF_DIR:
              $HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:
              $HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:
              $HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:
              $HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:
              /usr/lib/hadoop-lzo/lib/*:
              /usr/share/aws/emr/emrfs/conf:
              /usr/share/aws/emr/emrfs/lib/*:
              /usr/share/aws/emr/emrfs/auxlib/*:
              /usr/share/aws/emr/lib/*:
              /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:
              /usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:
              /usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:
              /usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:
              /usr/lib/spark/yarn/lib/datanucleus-core.jar:
              /usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:
              /usr/share/aws/emr/cloudwatch-sink/lib/*
           
      
      

      From yarn-site.xml:

      <property>
          <description>Classpath for typical applications.</description>
           <name>yarn.application.classpath</name>
           <value>
              $HADOOP_CONF_DIR,
              $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
              $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
              $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
              $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,
              /usr/lib/hadoop-lzo/lib/*,
              /usr/share/aws/emr/emrfs/conf,
              /usr/share/aws/emr/emrfs/lib/*,
              /usr/share/aws/emr/emrfs/auxlib/*,
              /usr/share/aws/emr/lib/*,
              /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar,
              /usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar,
              /usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar,
              /usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar,
              /usr/lib/spark/yarn/lib/datanucleus-core.jar,
              /usr/lib/spark/yarn/lib/datanucleus-rdbms.jar,
              /usr/share/aws/emr/cloudwatch-sink/lib/*
           </value>
        </property>
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: