Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-595

Refactor SparkRuntimeService and SparkProgramWrapper

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: Spark
    • Labels:
      None
    • Rank:
      1|hzyh2n:

      Description

      The current implementation involves unnecessary file copying and creation, as well as the organization is pretty messy, probably because of some code copied from the MR program runner, which they aren't the same.

      In MR case, we need to create job jar and other stuff since the MRRuntimeService is a MR client that submit job to YARN and YARN will create a new AM to the MR.

      In Spark case, however, the SparkRuntimeService is the driver itself (in both local and cluster mode), hence a lot of complications are not needed.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: