Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-9886

Hydrator python plugin has bad error messaging

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.1
    • Component/s: Pipelines
    • Labels:
      None
    • Rank:
      1|hzz5f3:

      Description

      I've attached a python script transform that has a bug in the script. The problem is that the output schema uses type long but the actual type is double.
      The error is buried in the container logs:

      2016-01-19 00:06:42,705 - ERROR [main:c.c.c.e.b.m.ETLMapReduce$ETLMapper@432] - Exception thrown in BatchDriver Mapper: {}
      java.lang.IllegalArgumentException: Could not transform input: null
      	at co.cask.hydrator.transforms.PythonEvaluator.transform(PythonEvaluator.java:177) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator.transform(PythonEvaluator.java:54) ~[na:na]
      	at co.cask.cdap.etl.common.LoggedTransform$2.call(LoggedTransform.java:57) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.LoggedTransform$2.call(LoggedTransform.java:54) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.log.LogContext.run(LogContext.java:54) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.LoggedTransform.transform(LoggedTransform.java:54) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.TrackedTransform.transform(TrackedTransform.java:43) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.TransformExecutor.executeTransformation(TransformExecutor.java:85) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.TransformExecutor.executeTransformation(TransformExecutor.java:90) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.common.TransformExecutor.runOneIteration(TransformExecutor.java:49) ~[cdap-etl-core-3.3.0-SNAPSHOT.jar:na]
      	at co.cask.cdap.etl.batch.mapreduce.ETLMapReduce$ETLMapper.map(ETLMapReduce.java:406) ~[1453161985610-0/:na]
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) [hadoop-mapreduce-client-core-2.6.0-cdh5.4.0.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapperWrapper.run(MapperWrapper.java:102) [co.cask.cdap.cdap-app-fabric-3.3.0-SNAPSHOT.jar:na]
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) [hadoop-mapreduce-client-core-2.6.0-cdh5.4.0.jar:na]
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) [hadoop-mapreduce-client-core-2.6.0-cdh5.4.0.jar:na]
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) [hadoop-mapreduce-client-app.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_75]
      	at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_75]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) [hadoop-common-2.6.0-cdh5.4.0.jar:na]
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) [hadoop-mapreduce-client-app.jar:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
      	at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerLauncher.launch(MapReduceContainerLauncher.java:91) [launcher.jar:na]
      	at org.apache.hadoop.mapred.YarnChild.main(Unknown Source) [hadoop-mapreduce-client-app.jar:na]
      Caused by: org.python.core.PyException: null
      	at org.python.core.Py.JavaError(Py.java:481) ~[na:na]
      	at org.python.core.Py.JavaError(Py.java:474) ~[na:na]
      	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188) ~[na:na]
      	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204) ~[na:na]
      	at org.python.core.PyObject.__call__(PyObject.java:404) ~[na:na]
      	at org.python.core.PyObject.__call__(PyObject.java:408) ~[na:na]
      	at org.python.core.PyMethod.__call__(PyMethod.java:124) ~[na:na]
      	at org.python.pycode._pyx0.transform$1(<script>:2) ~[na:na]
      	at org.python.pycode._pyx0.call_function(<script>) ~[na:na]
      	at org.python.core.PyTableCode.call(PyTableCode.java:165) ~[na:na]
      	at org.python.core.PyBaseCode.call(PyBaseCode.java:166) ~[na:na]
      	at org.python.core.PyFunction.__call__(PyFunction.java:338) ~[na:na]
      	at org.python.pycode._pyx0.f$0(<script>:9) ~[na:na]
      	at org.python.pycode._pyx0.call_function(<script>) ~[na:na]
      	at org.python.core.PyTableCode.call(PyTableCode.java:165) ~[na:na]
      	at org.python.core.PyCode.call(PyCode.java:18) ~[na:na]
      	at org.python.core.Py.runCode(Py.java:1261) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator.transform(PythonEvaluator.java:174) ~[na:na]
      	... 25 common frames omitted
      Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Long
      	at co.cask.hydrator.transforms.PythonEvaluator.decodeSimpleType(PythonEvaluator.java:303) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator.decode(PythonEvaluator.java:261) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator.decodeRecord(PythonEvaluator.java:286) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator.access$000(PythonEvaluator.java:57) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator$PythonEmitter.decode(PythonEvaluator.java:150) ~[na:na]
      	at co.cask.hydrator.transforms.PythonEvaluator$PythonEmitter.emit(PythonEvaluator.java:134) ~[na:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
      	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186) ~[na:na]
      	... 40 common frames omitted
      

      But all I was seeing in the logs from the UI was some null error. I can't attach a screenshot either because I navigate back to the page and now I don't see any logs.

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: