Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5267

PartitionFilter.ALWAYS_MATCH doesn't work as an input partition filter

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.2
    • Fix Version/s: 3.4.0
    • Component/s: Datasets
    • Labels:
      None
    • Release Notes:
      Hide
      PartitionFilter is now serialized into one key of the runtime arguments, in order to support serialization of PartitionFilter.ALWAYS_MATCH. Additionally, if there are additional fields in the PartitionFilter that do not exist in the partitioning, the filter will never match.
      Show
      PartitionFilter is now serialized into one key of the runtime arguments, in order to support serialization of PartitionFilter.ALWAYS_MATCH. Additionally, if there are additional fields in the PartitionFilter that do not exist in the partitioning, the filter will never match.
    • Rank:
      1|hzz8qv:

      Description

      I have some code in my beforeSubmit():

          Map<String, String> arguments = new HashMap<>();
          PartitionedFileSetArguments.setInputPartitionFilter(arguments, PartitionFilter.ALWAYS_MATCH);
          context.setInput(datasetName, arguments);
      

      setting the input partition filter to PartitionFilter.ALWAYS_MATCH is a no-op, and the arguments map stays empty. When the job is submitted, it then dies because it says no input paths are given. I would expect setting the always match filter to use all partitions as input.

      Not a critical bug, as the workaround is 3 lines longer:

          Map<String, String> arguments = new HashMap<>();
          PartitionedFileSet inputFileset = context.getDataset(datasetName);
          for (PartitionDetail partitionDetail : inputFileset.getPartitions(PartitionFilter.ALWAYS_MATCH)) {
            PartitionedFileSetArguments.addInputPartition(arguments, partitionDetail);
          }
          context.setInput(datasetName, arguments);
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                ashau Albert Shau
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: