Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2881

Support HDP 2.2.x

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: Distribution
    • Labels:
      None
    • Rank:
      1|hzyvc7:

      Description

      When trying to run a MR job on HDP 2.2.4.2, either via Hive query or MR program, the job submission always fails with the following error:

      2015-06-26 20:08:41,167 ERROR [HiveServer2-Background-Pool: Thread-263] exec.Task (SessionState.java:printError(833)) - Job Submission failed with exception 'java.lang.IllegalArgumentException(Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path)'
      java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
              at org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:739)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:435)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:585)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:580)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
              at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:580)
              at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:571)
              at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:429)
              at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
              at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
              at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
              at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
              at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1001)
              at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:153)
              at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
              at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:205)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
              at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
              at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:217)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.net.URISyntaxException: Illegal character in path at index 11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
              at java.net.URI$Parser.fail(URI.java:2829)
              at java.net.URI$Parser.checkChars(URI.java:3002)
              at java.net.URI$Parser.parseHierarchical(URI.java:3086)
              at java.net.URI$Parser.parse(URI.java:3044)
              at java.net.URI.<init>(URI.java:595)
              at org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:737)
              ... 36 more
      

      to fix, we need to pass -Dhdp.version=2.2 to the job submitter.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                alvin Alvin Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: