Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10301

DBSink Does not handle values with null even if the schema is marked as nullable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.2, 3.2.2, 3.0.6, 3.1.2
    • Fix Version/s: 3.4.0
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
      None
    • Release Notes:
      Fixed a null pointer exception when null values are written to a database sink in Hydrator.
    • Rank:
      1|hzz8rz:

      Description

      Create a pipeline with a nullable field and run the pipleline with some null values in the input. The pipeline fails with the following stacktrace. Please see attached config json

      BatchDriver Mapper.
      java.lang.NullPointerException: null
      	at co.cask.hydrator.plugin.DBRecord.writeToDB(DBRecord.java:327) ~[na:na]
      	at co.cask.hydrator.plugin.DBRecord.write(DBRecord.java:145) ~[na:na]
      	at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:118) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at co.cask.hydrator.plugin.db.batch.sink.ETLDBOutputFormat$1.write(ETLDBOutputFormat.java:109) ~[na:na]
      	at co.cask.hydrator.plugin.db.batch.sink.ETLDBOutputFormat$1.write(ETLDBOutputFormat.java:69) ~[na:na]
      	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.BasicMapReduceTaskContext.write(BasicMapReduceTaskContext.java:158) ~[co.cask.cdap.cdap-app-fabric-3.3.2.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceLifecycleContext.write(MapReduceLifecycleContext.java:67) ~[co.cask.cdap.cdap-app-fabric-3.3.2.jar:na]
      	at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce$SingleOutputSink.write(ETLMapReduce.java:491) ~[ETLWorkflow.a1fe5af9-4914-43c2-a69f-bcc896d60bdd/:na]
      	at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce$ETLMapper.map(ETLMapReduce.java:410) ~[ETLWorkflow.a1fe5af9-4914-43c2-a69f-bcc896d60bdd/:na]
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) [org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapperWrapper.run(MapperWrapper.java:102) [co.cask.cdap.cdap-app-fabric-3.3.2.jar:na]
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) [org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) [org.apache.hadoop.hadoop-mapreduce-client-core-2.3.0.jar:na]
      	at org.apache.hadoop.mapred.LocalJobRunnerWithFix$Job$MapTaskRunnable.run(LocalJobRunnerWithFix.java:243) [co.cask.cdap.cdap-app-fabric-3.3.2.jar:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
      	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
      
      

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: