Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7082

SparkPluginsTest fails with NoSuchMethod error.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 3.5.0
    • Component/s: App Fabric
    • Labels:
      None
    • Release Notes:
      Removed bundling of the parquet jar from com.twitter package with CDAP master.
    • Rank:
      1|hzzkcv:

      Description

      The error message we get is -

      parquet.schema.Types$MessageTypeBuilder.addFields([Lparquet/schema/Type;)Lparquet/schema/Types$BaseGroupBuilder;'
      	at co.cask.cdap.explore.client.ExploreHttpClient.nextResults(ExploreHttpClient.java:254)
      

      And corresponding exception is -

      java.lang.NoSuchMethodError: parquet.schema.Types$MessageTypeBuilder.addFields([Lparquet/schema/Type;)Lparquet/schema/Types$BaseGroupBuilder;
          at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.getSchemaByName(DataWritableReadSupport.java:158) ~[hive-exec-1.1.0-cdh5.7.0.jar:1.1.0-cdh5.7.0]
          at org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(DataWritableReadSupport.java:221) ~[hive-exec-1.1.0-cdh5.7.0.jar:1.1.0-cdh5.7.0]
          at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:256) ~[hive-exec-1.1.0-cdh5.7.0.jar:1.1.0-cdh5.7.0]
          at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95) ~[hive-exec-1.1.0-cdh5.7.0.jar:1.1.0-cdh5.7.0]
          at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81) ~[hive-exec-1.1.0-cdh5.7.0.jar:1.1.0-cdh5.7.0]
          at 
      

      parquet jar from com.twitter is getting picked up instead of hive.

      Fix is to exclude the dependencies for com.twitter:parquet-hadoop-bundle.

        Attachments

          Activity

            People

            • Assignee:
              sagar Sagar Kapare
              Reporter:
              sagar Sagar Kapare
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: