Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-8087

CDAP Workflow cannot fork multiple parallel Spark actions consistently

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0, 4.0.1, 3.5.3
    • Component/s: Spark, Workflow
    • Labels:
      None

      Description

      When the CDAP workflow driver encounters a fork action, it attempts to spark-submit in parallel threads. The stack trace below indicates this eventually leads to multiple parallel threads executing reflection code, which is unsafe in Scala 2.10.x. Please refer http://docs.scala-lang.org/overviews/reflection/thread-safety.html and https://issues.scala-lang.org/browse/SI-6240. The only way to prevent the problem is to synchronize the block that executes scala 2.10.x Reflection code.
      The exception stack trace is below.

       
      java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: tail of empty list 
      at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] 
      at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] 
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] 
      at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:268) ~[co.cask.cdap.cdap-spark-core-3.5.2-SNAPSHOT.jar:na] 
      at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] 
      at co.cask.cdap.app.runtime.spark.SparkRuntimeService$3$1.run(SparkRuntimeService.java:330) [co.cask.cdap.cdap-spark-core-3.5.2-SNAPSHOT.jar:na] 
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 
      Caused by: java.lang.UnsupportedOperationException: tail of empty list 
      at scala.collection.immutable.Nil$.tail(List.scala:339) ~[org.scala-lang.scala-library-2.10.4.jar:na] 
      at scala.collection.immutable.Nil$.tail(List.scala:334) ~[org.scala-lang.scala-library-2.10.4.jar:na] 
      at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:172) ~[spark-assembly.jar:na] 
      at scala.reflect.internal.Symbols$Symbol.unsafeTypeParams(Symbols.scala:1477) ~[spark-assembly.jar:na] 
      at scala.reflect.internal.Symbols$TypeSymbol.tpe(Symbols.scala:2777) ~[spark-assembly.jar:na] 
      at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:235) ~[spark-assembly.jar:na] 
      at scala.reflect.runtime.JavaMirrors$class.createMirror(JavaMirrors.scala:34) ~[spark-assembly.jar:na] 
      at scala.reflect.runtime.JavaMirrors$class.runtimeMirror(JavaMirrors.scala:61) ~[spark-assembly.jar:na] 
      at scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:12) ~[spark-assembly.jar:na] 
      at scala.reflect.runtime.JavaUniverse.runtimeMirror(JavaUniverse.scala:12) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1256) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:276) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:648) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:124) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:927) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:986) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.yarn.Client.main(Client.scala) ~[spark-assembly.jar:na] 
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] 
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] 
      at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] 
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) ~[spark-assembly.jar:na] 
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-assembly.jar:na] 
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.submit(AbstractSparkSubmitter.java:172) ~[co.cask.cdap.cdap-spark-core-3.5.2-SNAPSHOT.jar:na] 
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter.access$000(AbstractSparkSubmitter.java:56) ~[co.cask.cdap.cdap-spark-core-3.5.2-SNAPSHOT.jar:na] 
      at co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter$5.run(AbstractSparkSubmitter.java:114) ~[co.cask.cdap.cdap-spark-core-3.5.2-SNAPSHOT.jar:na] 
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60] 
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60] 
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60] 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60]
      

      The FATAL 'tail of empty list' exception occurs only when the CDAP workflow folks to spawn multiple Spark jobs in parallel.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: