Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10394

DBSource fails in HDP Cluster because the JDBC Driver class is not found when trying to submit the MR Job

    Details

    • Release Notes:
      Fixed a problem with Hydrator pipelines using a DBSource not working in an HDP cluster.
    • Rank:
      1|hzzdtj:

      Description

      Created a HDP cluster and created a DBSource -> TPFS batch (MR) pipeline and it fails with the following issue:

      2016-05-27 19:35:29,409 - ERROR [MapReduceRunner-phase-1:c.c.c.i.a.r.b.MapReduceRuntimeService@299] - Exception when submitting MapReduce Job: job=phase-1,=namespaceId=default, applicationId=D2TAvro3, program=phase-1, runid=22b8d9c1-2442-11e6-9138-42010a800009
      java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
              at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:171) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) ~[hadoop-common-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) ~[hadoop-common-2.7.1.2.3.4.7-4.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.dataset.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:69) ~[co.cask.cdap.cdap-app-fabric-3.4.1.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_75]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:290) ~[co.cask.cdap.cdap-app-fabric-3.4.1.jar:na]
              at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:386) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
              at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.createConnection(DBInputFormat.java:205) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:164) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              ... 16 common frames omitted
      Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_75]
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_75]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_75]
              at java.lang.Class.forName0(Native Method) ~[na:1.7.0_75]
              at java.lang.Class.forName(Class.java:191) ~[na:1.7.0_75]
              at org.apache.hadoop.mapreduce.lib.db.DBConfiguration.getConnection(DBConfiguration.java:148) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.createConnection(DBInputFormat.java:198) ~[hadoop-mapreduce-client-core-2.7.1.2.3.4.7-4.jar:na]
              ... 17 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              shankar Shankar Selvam
              Reporter:
              gokul Gokul Gunasekaran
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: