Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.2.0, 6.1.2
    • Component/s: Pipeline Plugins
    • Labels:
      None
    • Release Notes:
      Fixed macro support for output schema in BQ source plugin
    • Rank:
      1|i00uof:

      Description

      BigQuery source plugin supports macros, but it fails with the following stack trace when output schema is a macro.

      io.cdap.cdap.api.data.format.UnexpectedFormatException: Schema must be a record with at least one field.
      	at io.cdap.cdap.api.data.format.StructuredRecord.builder(StructuredRecord.java:281) ~[na:na]
      	at io.cdap.plugin.gcp.bigquery.source.BigQueryAvroToStructuredTransformer.transform(BigQueryAvroToStructuredTransformer.java:40) ~[1584053955151-0/:na]
      	at io.cdap.plugin.gcp.bigquery.source.BigQuerySource.transform(BigQuerySource.java:192) ~[1584053955151-0/:na]
      	at io.cdap.cdap.etl.common.plugin.WrappedBatchSource.lambda$transform$2(WrappedBatchSource.java:69) ~[cdap-etl-core-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.common.plugin.Caller$1.call(Caller.java:30) ~[cdap-etl-core-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.common.plugin.WrappedBatchSource.transform(WrappedBatchSource.java:68) ~[cdap-etl-core-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.common.plugin.WrappedBatchSource.transform(WrappedBatchSource.java:36) ~[cdap-etl-core-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.common.TrackedTransform.transform(TrackedTransform.java:74) ~[cdap-etl-core-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.spark.function.BatchSourceFunction.call(BatchSourceFunction.java:54) ~[hydrator-spark-core2_2.11-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.spark.function.BatchSourceFunction.call(BatchSourceFunction.java:32) ~[hydrator-spark-core2_2.11-6.2.0-SNAPSHOT.jar:na]
      	at io.cdap.cdap.etl.spark.Compat$FlatMapAdapter.call(Compat.java:126) ~[hydrator-spark-core2_2.11-6.2.0-SNAPSHOT.jar:na]
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1123) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1356) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.scheduler.Task.run(Task.scala:100) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_221]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_221]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_221]
      

        Attachments

          Activity

            People

            • Assignee:
              rmstar Prashant Jaikumar
              Reporter:
              rmstar Prashant Jaikumar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: