Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7528

SparkKMeansAppTest on CDAP VM fails

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.5.1
    • Fix Version/s: None
    • Component/s: CDAP Examples, Test
    • Labels:
    • Rank:
      1|hzznrb:

      Description

      On the CDAP Standalone VM, executing the SparkKMeansAppTest results in the following errors:

      2016-10-26 04:32:21,031 - INFO [main:c.c.c.a.r.s.SparkProgramRunner@196] - Starting Spark Job: SparkRuntimeContext{id=program:default.SparkKMeans.spark.SparkKMeansProgram, runId=2efad022-9b35-11e6-ba65-080027e799c9} 
      2016-10-26 04:32:22,776 - WARN [spark-submitter-SparkKMeansProgram-2efad022-9b35-11e6-ba65-080027e799c9:o.a.s.Logging$class@70] - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN). 
      2016-10-26 04:32:22,824 - WARN [spark-submitter-SparkKMeansProgram-2efad022-9b35-11e6-ba65-080027e799c9:o.a.s.Logging$class@70] - Your hostname, cdap-standalone-vm-3 resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0) 
      2016-10-26 04:32:22,824 - WARN [spark-submitter-SparkKMeansProgram-2efad022-9b35-11e6-ba65-080027e799c9:o.a.s.Logging$class@70] - Set SPARK_LOCAL_IP if you need to bind to another address 
      2016-10-26 04:32:28,925 - WARN [Executor task launch worker-2:c.g.f.n.BLAS@61] - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 
      2016-10-26 04:32:28,926 - WARN [Executor task launch worker-2:c.g.f.n.BLAS@61] - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 
      2016-10-26 04:32:39,972 - WARN [driver-heartbeater:o.a.s.Logging$class@91] - Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@719d1ebc,BlockManagerId(driver, localhost, 41983))] in 1 attempts
      
      2016-10-26 04:32:45,977 - WARN [driver-heartbeater:o.a.s.Logging$class@91] - Issue communicating with driver in heartbeater 
      org.apache.spark.SparkException: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@719d1ebc,BlockManagerId(driver, localhost, 41983))] 
      at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118) ~[spark-core_2.10-1.6.1.jar:1.6.1] 
      at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449) ~[spark-core_2.10-1.6.1.jar:1.6.1] 
      at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470) [spark-core_2.10-1.6.1.jar:1.6.1] 
      at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [spark-core_2.10-1.6.1.jar:1.6.1] 
      at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [spark-core_2.10-1.6.1.jar:1.6.1] 
      at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765) [spark-core_2.10-1.6.1.jar:na] 
      at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470) [spark-core_2.10-1.6.1.jar:1.6.1] 
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] 
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_75] 
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_75] 
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_75] 
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75] 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75] 
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
      

      Increasing VM memory to 8GB seems to resolve the issue.

        Attachments

          Activity

            People

            • Assignee:
              nitin Nitin Motgi
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: