Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6170

Apps should be able to override some dataset properties at runtime

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.2
    • Fix Version/s: 3.5.0
    • Component/s: Datasets
    • Labels:
      None
    • Release Notes:
      Fixed an issue that caused an error if an application/program attempted to override input/output format properties that were already defined in the dataset properties.
    • Rank:
      1|hzze7z:

      Description

      For example, suppose a FileSet is declared with a schema. An app should be able to override that with a projection of that schema, especially to drop fields from the schema.

      Currently, if a FileSet is created with a schema in the input format configuration, and the program attempts to set a different output schema, it results in the following:

      Caused by: java.lang.IllegalArgumentException: duplicate key: avro.schema.output.key at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.collect.ImmutableMap$Builder.fromEntryList(ImmutableMap.java:249) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:235) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.data2.dataset2.lib.file.FileSetDataset.getOutputFormatConfiguration(FileSetDataset.java:245) ~[co.cask.cdap.cdap-data-fabric-3.5.0-SNAPSHOT.jar:na] at co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset.getOutputFormatConfiguration(PartitionedFileSetDataset.java:725) ~[co.cask.cdap.cdap-data-fabric-3.5.0-SNAPSHOT.jar:na] at co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext$$anon$2.run(DefaultSparkExecutionContext.scala:216) ~[na:na] at co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:197) ~[na:na] ... 29 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              andreas Andreas Neumann
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: