Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14106

SparkPageRank example's Spark program fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 5.1.0
    • Fix Version/s: 5.1.0
    • Component/s: CDAP Examples
    • Labels:
      None
    • Rank:
      1|i00ghz:

      Description

      The failure is probably due to switching Spark versions to Spark 2.
      I deployed the example SparkPageRank, ingested some events into its stream, and then ran theĀ PageRankSpark program. It fails, with the following error message:

      2018-08-13 20:30:37,503 - ERROR [SparkRunnerPageRankSpark:c.c.c.i.a.r.ProgramControllerServiceAdapter@97] - Spark Program 'PageRankSpark' failed.
      java.util.concurrent.ExecutionException: org.apache.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 5, localhost, executor driver): java.lang.AbstractMethodError: co.cask.cdap.examples.sparkpagerank.SparkPageRankProgram$3.call(Ljava/lang/Object;)Ljava/util/Iterator;
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
      	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
      	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
      	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
      	at org.apache.spark.scheduler.Task.run(Task.scala:99)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Driver stacktrace:
      	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
      	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]
      	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:347) ~[co.cask.cdap.cdap-spark-core2_2.11-5.1.0-SNAPSHOT.jar:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.app.runtime.spark.SparkRuntimeService$5$1.run(SparkRuntimeService.java:405) [co.cask.cdap.cdap-spark-core2_2.11-5.1.0-SNAPSHOT.jar:na]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
      Caused by: org.apache.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 5, localhost, executor driver): java.lang.AbstractMethodError: co.cask.cdap.examples.sparkpagerank.SparkPageRankProgram$3.call(Ljava/lang/Object;)Ljava/util/Iterator;
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
      	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
      	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
      	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
      	at org.apache.spark.scheduler.Task.run(Task.scala:99)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748) 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: