Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7429

hdp 2.5 spark support detection

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: CDAP
    • Labels:
      None
    • Release Notes:
      Fixed an issue with Spark detection on HDP 2.5 and above, causing excess noise on the console.
    • Rank:
      1|hzzmzz:

      Description

      HDP 2.5 has changed some packaging around spark that makes CDAP's spark detection noisy. The hdp-select package now provides /usr/bin/spark-shell, which then may fail in their custom wrapper script. This leads to the following ugly error message on startup:

      [root@hdp25-14361-1000 conf]# /etc/init.d/cdap-master start
      ERROR - While determining Spark home, failed to get Spark settings using: spark-shell --master local
      stderr:
      spark-shell is not found, please check if spark is installed
      /usr/bin/spark-shell: line 26: exec: : not found
      Could not determine SPARK_HOME! Spark support unavailable!
      

      CDAP's detection should be updated to handle this.

        Attachments

          Activity

            People

            • Assignee:
              chris Chris Gianelloni
              Reporter:
              derek Derek Wood
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: