Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2945

MapReduce job using FileSet/PartitionedFileSet as input fails if there are no input partitions.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.6.0, 3.5.1, 3.5.0, 3.4.0, 3.3.0, 3.2.0, 3.1.0, 3.0.1, 3.0.0
    • Fix Version/s: 4.0.0, 3.6.1, 3.5.2
    • Component/s: CDAP, Datasets, MapReduce
    • Labels:
      None
    • Release Notes:
      MapReduce job using FileSet/PartitionedFileSet as input no longer fail if there are no input partitions.
    • Rank:
      1|hzyvov:

      Description

      If a PartitionedFileSet dataset is used as the input for a MapReduce job and a PartitionFilter is specified that matches zero partitions, the MapReduce job will throw an exception:

      java.lang.RuntimeException: java.io.IOException: No input paths specified in job
      	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:301) ~[classes/:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [classes/:na]
      	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
      Caused by: java.io.IOException: No input paths specified in job
      	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231) ~[hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340) ~[hadoop-mapreduce-client-core-2.3.0.jar:na]
      

      This is the behavior in Hadoop MapReduce (if no input path is specified for FileInputFormat).

      A possibility to consider is to use a temp file as the input, to at least create an empty partition, if no input partitions are matched/specified.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: