Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6783

Spark programs fail to start

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 3.5.0
    • Component/s: Spark
    • Labels:
    • Release Notes:
      Fixed the classpath ordering in Spark to load the classes from the cdap-common.jar at the beginning.
    • Rank:
      1|hzzi53:

      Description

      I'm getting issues starting Spark programs. In stderr, I see:

      Exception in thread "main" java.lang.RuntimeException: Failed to call org.apache.spark.deploy.yarn.ApplicationMaster.main(String[])
      	at co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch(SparkContainerLauncher.java:67)
      	at org.apache.spark.deploy.yarn.ApplicationMaster.main(Unknown Source)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch(SparkContainerLauncher.java:65)
      	... 1 more
      Caused by: java.lang.NoSuchMethodError: org.apache.twill.filesystem.FileContextLocationFactory.<init>(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/FileContext;Ljava/lang/String;)V
      	at co.cask.cdap.common.guice.RootLocationFactoryProvider.get(RootLocationFactoryProvider.java:44)
      	at co.cask.cdap.common.guice.RootLocationFactoryProvider.get(RootLocationFactoryProvider.java:31)
      	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      	at com.google.inject.Scopes$1$1.get(Scopes.java:65)
      	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
      	at com.google.inject.internal.ExposedKeyFactory.get(ExposedKeyFactory.java:54)
      	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
      	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
      	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
      	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
      	at com.google.inject.internal.ExposedKeyFactory.get(ExposedKeyFactory.java:54)
      	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
      	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
      	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
      	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
      	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
      	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
      	at com.google.inject.Scopes$1$1.get(Scopes.java:65)
      	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
      	at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
      	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
      	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
      	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.createIfNotExists(SparkRuntimeContextProvider.java:173)
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider.get(SparkRuntimeContextProvider.java:98)
      	at co.cask.cdap.app.runtime.spark.SparkClassLoader.create(SparkClassLoader.java:69)
      	at org.apache.spark.util.MutableURLClassLoader.createParent(MutableURLClassLoader.java:60)
      	at org.apache.spark.util.MutableURLClassLoader.<init>(MutableURLClassLoader.java:39)
      	at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:524)
      	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:319)
      	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:185)
      	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:653)
      	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
      	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
      	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
      	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:651)
      	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
      	... 6 more
      

      Looks like the twill jar is getting picked up before the cdap one because in launch_container.sh:

      export CLASSPATH="logback.xml5238129696552851943.jar:$PWD/cdap-spark-launcher.jar:$PWD/cdap-spark.jar/lib/*:$PWD:$PWD/__spark_conf__:$PWD/__spark__.jar:$HADOOP_CLIENT_CONF_DIR:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$MR2_CLASSPATH:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$MR2_CLASSPATH"
      

      And the cdap-spark.jar/lib/* doesn't guarantee any ordering.

        Attachments

          Activity

            People

            • Assignee:
              sagar Sagar Kapare
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: