Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13230

Creating data pipeline from data prep for parsing avro fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1.0
    • Component/s: Data Prep, Pipelines
    • Labels:
    • Release Notes:
      Fixed a bug that caused DataPrep fail while parsing avro files.
    • Rank:
      1|i00bjz:

      Description

      While creating a pipelines from data prep after using the directive parse-as-avro

      parse-as-avro :body exampleschema binary
      

      Commands to register in schema registry

      curl -X PUT "http://localhost:11015/v3/namespaces/default/apps/dataprep/services/service/methods/schemas?id=exampleschema&name=testschema&description=testwqwwq&type=avro"
      
      curl -X POST --data-binary @120.avsc "http://localhost:11015/v3/namespaces/default/apps/dataprep/services/service/methods/schemas/exampleschema"
      

      fails with the following exception

      2018-03-22 19:00:22,381 - WARN  [Thread-121:o.a.h.m.LocalJobRunner$Job@587] - job_local788673871_0003
      java.lang.Exception: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -49
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
      org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -49
      	at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:363) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:355) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) ~[avro-1.7.7.jar:1.7.7]
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) ~[avro-1.7.7.jar:1.7.7]
      	at co.cask.wrangler.codec.BinaryAvroDecoder.decode(BinaryAvroDecoder.java:67) ~[wrangler-core-3.0.1.jar:na]
      	at co.cask.directives.parser.ParseAvro.execute(ParseAvro.java:172) ~[na:na]
      	at co.cask.directives.parser.ParseAvro.execute(ParseAvro.java:62) ~[na:na]
      	at co.cask.wrangler.executor.RecipePipelineExecutor.execute(RecipePipelineExecutor.java:121) ~[wrangler-core-3.0.1.jar:na]
      	at co.cask.wrangler.executor.RecipePipelineExecutor.execute(RecipePipelineExecutor.java:96) ~[wrangler-core-3.0.1.jar:na]
      	at co.cask.wrangler.Wrangler.transform(Wrangler.java:339) ~[1521770408472-0/:na]
      	at co.cask.wrangler.Wrangler.transform(Wrangler.java:74) ~[1521770408472-0/:na]
      	at co.cask.cdap.etl.common.plugin.WrappedTransform$6.call(WrappedTransform.java:107) ~[cdap-etl-core-4.3.1.jar:na]
      	at co.cask.cdap.etl.common.plugin.WrappedTransform$6.call(WrappedTransform.java:104) ~[cdap-etl-core-4.3.1.jar:na]
      

        Attachments

        1. 120.avsc
          0.5 kB
        2. 120.frag.avro
          0.8 kB
        3. 120.json
          2 kB
        4. Test_v1-cdap-data-pipeline.json
          8 kB

          Issue Links

            Activity

              People

              • Assignee:
                ashau Albert Shau
                Reporter:
                sree Sreevatsan Raman
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: