Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5019

Fail to launch Spark job on Cloudera Manager cluster with Spark 1.4+

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.1, 3.2.2
    • Fix Version/s: 3.3.2
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|hzz77j:

      Description

      Root cause is caused by the NPE from Spark.

      https://issues.apache.org/jira/browse/SPARK-13441

      It only happens in CM cluster because CM use a private directory as the HADOOP_CONF_DIR for the NodeManager, which the NodeManager always use it as the container env variable HADOOP_CONF_DIR.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: