Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13026

Having Parquet snapshot as source and sink throws an error on deploying the pipeline

    Details

    • Release Notes:
      Fixed a bug that prevented a parquet snapshot source and sink to be used in the same pipeline
    • Rank:
      1|i00acn:

      Description

      Having a Snapshot parquet dataset as source and sink fails during the deployment with the following error

      Caused by: java.lang.RuntimeException: Exception while configuring plugin of type batchsink and name TPFSParquet for stage Parquet Time Partitioned Dataset: DatasetInstance 'ParquetSanapShot' was added multiple times with different specifications. Please resolve the conflict so that there is only one specification for the dataset instance.
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configurePlugin(PipelineSpecGenerator.java:348) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:257) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configureStages(PipelineSpecGenerator.java:147) ~[na:na]
      	at co.cask.cdap.etl.batch.BatchPipelineSpecGenerator.generateSpec(BatchPipelineSpecGenerator.java:56) ~[na:na]
      	at co.cask.cdap.etl.batch.BatchPipelineSpecGenerator.generateSpec(BatchPipelineSpecGenerator.java:34) ~[na:na]
      	at co.cask.cdap.datapipeline.DataPipelineApp.configure(DataPipelineApp.java:80) ~[na:na]
      	at co.cask.cdap.api.app.AbstractApplication.configure(AbstractApplication.java:65) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.getSpecJson(InMemoryConfigurator.java:160) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.createResponse(InMemoryConfigurator.java:126) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.config(InMemoryConfigurator.java:116) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:108) ~[na:na]
      	... 20 common frames omitted
      Caused by: java.lang.IllegalArgumentException: DatasetInstance 'ParquetSanapShot' was added multiple times with different specifications. Please resolve the conflict so that there is only one specification for the dataset instance.
      	at co.cask.cdap.internal.api.DefaultDatasetConfigurer.createDataset(DefaultDatasetConfigurer.java:154) ~[na:na]
      	at co.cask.cdap.etl.common.DefaultPipelineConfigurer.createDataset(DefaultPipelineConfigurer.java:80) ~[na:na]
      	at co.cask.hydrator.plugin.batch.sink.TimePartitionedFileSetSink.configurePipeline(TimePartitionedFileSetSink.java:64) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configurePlugin(PipelineSpecGenerator.java:344) ~[na:na]
      	... 30 common frames omitted
      
      

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: