Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14346

Exploring parquet fileset with timestamp throws ClassCastException

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.1.0
    • Fix Version/s: None
    • Labels:
      None
    • Rank:
      1|i00hs7:

      Description

      In 5.1, we have added support for timestamp. For parquet fileset plugins, we use `AvroParquetOutputFormat` to write data. However, hive uses `org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat` to read parquet data when queried from explore. `MapredParquetInputFormat` only converts INT96 to timestamp. https://github.com/apache/hive/blob/release-1.2.1/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java#L181

      However, `AvroParquetOutputFormat` converts timestamp logical type to long(INT64). https://github.com/apache/parquet-mr/blob/apache-parquet-1.10.0/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L297

      Because of this discrepancy, below exception is thrown:

      java.lang.RuntimeException: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apa
      che.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
              at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.fetchNextResults(BaseHiveExploreService.java:1011) ~[na:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.previewResults(BaseHiveExploreService.java:1049) ~[na:na]
              at co.cask.cdap.explore.executor.ExploreQueryExecutorHttpHandler$5.call(ExploreQueryExecutorHttpHandler.java:213) ~[na:na]
              at co.cask.cdap.explore.executor.ExploreQueryExecutorHttpHandler$5.call(ExploreQueryExecutorHttpHandler.java:210) ~[na:na]
              at co.cask.cdap.security.impersonation.ImpersonationUtils$1.run(ImpersonationUtils.java:47) ~[na:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_151]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_151]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[org.apache.hadoop.hadoop-common-2.8.0.jar:na]
              at co.cask.cdap.security.impersonation.ImpersonationUtils.doAs(ImpersonationUtils.java:44) ~[na:na]
              at co.cask.cdap.explore.executor.ExploreQueryExecutorHttpHandler.doAs(ExploreQueryExecutorHttpHandler.java:71) ~[na:na]
              at co.cask.cdap.explore.executor.ExploreQueryExecutorHttpHandler.getQueryResultPreview(ExploreQueryExecutorHttpHandler.java:210) ~[na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
              at co.cask.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:82) ~[co.cask.http.netty-http-1.1.0-SNAPSHOT.jar:na]
              at co.cask.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) [co.cask.http.netty-http-1.1.0-SNAPSHOT.jar:na]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
              at co.cask.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) [co.cask.http.netty-http-1.1.0-SNAPSHOT.jar:na]
              at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Fina
      l]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_151]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
              at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
      Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.Lon
      gWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352) ~[org.apache.hive.hive-service-1.2.1.jar:1.2.1]
              at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:220) ~[org.apache.hive.hive-service-1.2.1.jar:1.2.1]
              at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685) ~[org.apache.hive.hive-service-1.2.1.jar:1.2.1]
              at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454) ~[org.apache.hive.hive-service-1.2.1.jar:1.2.1]
              at co.cask.cdap.explore.service.hive.Hive14ExploreService.doFetchNextResults(Hive14ExploreService.java:111) ~[na:na]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.fetchNextResults(BaseHiveExploreService.java:1006) ~[na:na]
              ... 29 common frames omitted
      Caused by: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:154) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1670) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347) ~[org.apache.hive.hive-service-1.2.1.jar:1.2.1]
              ... 34 common frames omitted
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
              at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:93) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:54) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:424) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:416) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              ... 36 common frames omitted
      Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
              at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.getPrimitiveJavaObject(WritableTimestampObjectInspector.java:39) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.getPrimitiveJavaObject(WritableTimestampObjectInspector.java:25) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:168) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.FetchFormatter$ThriftFormatter.convert(FetchFormatter.java:61) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:90) ~[org.apache.hive.hive-exec-1.2.1.jar:1.2.1]
              ... 45 common frames omitted 

        Attachments

          Activity

            People

            • Assignee:
              priyanka Priyanka Nambiar
              Reporter:
              vinisha Vinisha Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: