Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6378

MapReduce program sometimes fails with java.lang.NoSuchMethodError exception.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 3.5.0
    • Component/s: CDAP
    • Labels:
      None
    • Release Notes:
      Fixed the classpath of the MapReduce program launched by CDAP, to include CDAP classes before Apache Twill classes.
    • Rank:
      1|hzzfgn:

      Description

      MapReduce program fails with following exception -

      2016-07-07 19:36:13,789 - WARN  [main:o.a.h.m.YarnChild@173] - Exception running child : co.cask.cdap.api.data.DatasetInstantiationException: Could not instantiate dataset 'default:purchaseSource'
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache.getDataset(SingleThreadDatasetCache.java:179)
              at co.cask.cdap.data2.dataset2.DynamicDatasetCache.getDataset(DynamicDatasetCache.java:208)
              at co.cask.cdap.data2.dataset2.DynamicDatasetCache.getDataset(DynamicDatasetCache.java:182)
              at co.cask.cdap.data2.dataset2.DynamicDatasetCache.getDataset(DynamicDatasetCache.java:153)
              at co.cask.cdap.internal.app.runtime.AbstractContext.getDataset(AbstractContext.java:195)
              at co.cask.cdap.internal.app.runtime.batch.BasicMapReduceTaskContext.getDataset(BasicMapReduceTaskContext.java:246)
              at co.cask.cdap.internal.app.runtime.batch.BasicMapReduceTaskContext.getBatchReadable(BasicMapReduceTaskContext.java:289)
              at co.cask.cdap.internal.app.runtime.batch.MapReduceBatchReadableInputFormat.createBatchReadable(MapReduceBatchReadableInputFormat.java:39)
              at co.cask.cdap.internal.app.runtime.batch.dataset.AbstractBatchReadableInputFormat.createRecordReader(AbstractBatchReadableInputFormat.java:110)
              at co.cask.cdap.internal.app.runtime.batch.dataset.input.DelegatingRecordReader.<init>(DelegatingRecordReader.java:53)
              at co.cask.cdap.internal.app.runtime.batch.dataset.input.DelegatingInputFormat.createRecordReader(DelegatingInputFormat.java:80)
              at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:524)
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)
              at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
              at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerLauncher.launch(MapReduceContainerLauncher.java:98)
              at org.apache.hadoop.mapred.YarnChild.main(Unknown Source)
      Caused by: java.lang.NoSuchMethodError: org.apache.twill.filesystem.FileContextLocationFactory.<init>(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/FileContext;Ljava/lang/String;)V
              at co.cask.cdap.common.guice.LocationRuntimeModule$HDFSLocationModule.providesLocationFactory(LocationRuntimeModule.java:102)
      

      We copied the FileContextLocationFactory from Twill to CDAP and added one 3 arguments constructor. However sometimes MapReduce program picks up FileContextLocationFactory from Twill causing the above exception.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sagar Sagar Kapare
                Reporter:
                sagar Sagar Kapare
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: