Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12769

Dynamic spark program plugin fails on the Windows.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.3.1
    • Fix Version/s: 4.3.2
    • Component/s: Spark
    • Labels:
    • Release Notes:
      Fixed an issue with running the dynamic scala spark plugin on Windows. Directory which is used to store the compiled scala classes now contains '.' as a separator instead of ':' which was causing failure on Windows machines.
    • Rank:
      1|i008wv:

      Description

      Dynamic scala spark plugin fails on the windows with following exception

      Caused by: java.io.IOException: Failed to create directory C:\Users\sg0225378\AppData\Roaming\Cask\CDAPSandbox\data\tmp\program_run:default.SPDpndncy_v1.-SNAPSHOT.spark.phase-1.30b26fd5-bd3b-11e7-9e10-000000d73a92-classes-1 for storing compiled class files.
              at co.cask.cdap.app.runtime.spark.AbstractSparkExecutionContext.createInterpreter(AbstractSparkExecutionContext.scala:220) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SerializableSparkExecutionContext.createInterpreter(SerializableSparkExecutionContext.scala:66) ~[na:na]
              at co.cask.cdap.app.runtime.spark.DefaultJavaSparkExecutionContext.createInterpreter(DefaultJavaSparkExecutionContext.scala:100) ~[na:na]
              at co.cask.hydrator.plugin.spark.dynamic.ScalaSparkProgram.run(ScalaSparkProgram.java:94) ~[na:na]
              at co.cask.cdap.etl.spark.plugin.WrappedJavaSparkMain$1.call(WrappedJavaSparkMain.java:43) ~[na:na]
              at co.cask.cdap.etl.spark.plugin.WrappedJavaSparkMain$1.call(WrappedJavaSparkMain.java:40) ~[na:na]
              at co.cask.cdap.etl.common.plugin.Caller$1.call(Caller.java:30) ~[na:na]
              at co.cask.cdap.etl.common.plugin.StageLoggingCaller.call(StageLoggingCaller.java:40) ~[na:na]
              at co.cask.cdap.etl.spark.plugin.WrappedJavaSparkMain.run(WrappedJavaSparkMain.java:40) ~[na:na]
              at co.cask.cdap.datapipeline.JavaSparkMainWrapper.run(JavaSparkMainWrapper.java:60) ~[na:na]
      

      The reason for the failure is part `program_run:default.SPDpndncy_v1.-SNAPSHOT` in the path. Colon (':') is reserved character in Windows and is not allowed in the file/directory names.

      This line is causing issue - https://github.com/caskdata/cdap/blob/release/4.3/cdap-spark-core-base/src/main/scala/co/cask/cdap/app/runtime/spark/AbstractSparkExecutionContext.scala#L218 where it adds the toString() version of the program run id which gets colon in it.

        Attachments

          Activity

            People

            • Assignee:
              sagar Sagar Kapare
              Reporter:
              sagar Sagar Kapare
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: