CDAP
  1. CDAP
  2. CDAP-2952

Plugin avro library not accessible to MapReduce

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|hzyvpz:

      Description

      Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state -
      ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception
      java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0
      at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na]
      at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

      -------------------------------------------------------------------------------------------------------------------------------

      -------------------------------------------------------------------------------------------------------------------------------
      But digging into logs based on application id and run id on port 8088 tells me that the error is:
      2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381)
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961)
      at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461)
      ... 8 more
      Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959)
      ... 10 more

        Activity

        Hide
        Albert Shau added a comment -

        I think this has something to do with the FilterClassLoader changes I made in CDAP-2208. Reverting that change seems to fix the problem, but why...

        Show
        Albert Shau added a comment - I think this has something to do with the FilterClassLoader changes I made in CDAP-2208 . Reverting that change seems to fix the problem, but why...
        Hide
        Albert Shau added a comment -

        for whatever reason, the classloader change makes it so avro-mapred-1.7.7-hadoop2.jar is not getting placed in the job.jar/lib/ directory for the mapreduce run

        Show
        Albert Shau added a comment - for whatever reason, the classloader change makes it so avro-mapred-1.7.7-hadoop2.jar is not getting placed in the job.jar/lib/ directory for the mapreduce run
        Hide
        Albert Shau added a comment - - edited

        actually, it shouldn't be in there at all, so the old buggy behavior was covering up another bug where the plugin classloader is not being used in the mapreduce job, at least when Hadoop is trying to get the OutputCommitter. (this is on a cdh5.3.1 cluster).

        Show
        Albert Shau added a comment - - edited actually, it shouldn't be in there at all, so the old buggy behavior was covering up another bug where the plugin classloader is not being used in the mapreduce job, at least when Hadoop is trying to get the OutputCommitter. (this is on a cdh5.3.1 cluster).
        Hide
        Albert Shau added a comment -

        rohan was using an hdp-2.4.0.2 cluster

        Show
        Albert Shau added a comment - rohan was using an hdp-2.4.0.2 cluster
        Show
        Albert Shau added a comment - https://issues.apache.org/jira/browse/MAPREDUCE-5957
        Hide
        Terence Yim added a comment -
        Show
        Terence Yim added a comment - Fix in https://github.com/caskdata/cdap/pull/3140

          People

          • Assignee:
            Terence Yim
            Reporter:
            Rohan Nahata
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: