Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2952

Plugin avro library not accessible to MapReduce

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|hzyvpz:

      Description

      Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state -
      ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception
      java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0
      at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na]
      at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
      at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

      -------------------------------------------------------------------------------------------------------------------------------

      -------------------------------------------------------------------------------------------------------------------------------
      But digging into logs based on application id and run id on port 8088 tells me that the error is:
      2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381)
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961)
      at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461)
      ... 8 more
      Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959)
      ... 10 more

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              rohanahata Rohan Nahata
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: