Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-16309

Spark program failed with unsupported program type: Spark even Spark is present

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.2.0, 6.1.2
    • Component/s: App Fabric
    • Labels:
      None
    • Release Notes:
      Fixed a race condition bug that can causes failure when running Spark program
    • Rank:
      1|i00twv:

      Description

      In RemoteExecutionTwillPreparer, on startup of a program, it traces dependencies to build an application jar that contains CDAP+dep jars inside. It uses the context classloader to trace dependencies. However, since startup is async, and the RemoteExecution twill service uses a Thread pool to execute the startup, the context classloader from previous run may be used if the thread in the thread pool is still around (60 seconds idle time). If the previous run was a non-Spark program, it will be using a Classloader that doesn't contain SparkProgramRunner to trace dependencies, hence resulting in missing cdap-spark-core jar inside.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: