Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12731

HBase Sink classloader failure in Spark

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.1, 4.3.0
    • Fix Version/s: 5.0.0, 4.3.2
    • Component/s: Pipeline Plugins, Spark
    • Labels:
      None
    • Release Notes:
      Fixed an issue with that caused hbase sink to fail when used alongside other sinks, using spark execution engine.
    • Rank:
      1|i008p3:

      Description

      Attached a pipeline that encounters this failure.

      2017-10-18 00:41:35,434 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@95] - User class threw exception: co.cask.cdap.api.data.DatasetInstantiationException: Failed to access dataset: dataset:def
      ault.<dataset-name>
      co.cask.cdap.api.data.DatasetInstantiationException: Failed to access dataset: dataset:<dataset-name>
              at co.cask.cdap.data.dataset.SystemDatasetInstantiator.getDataset(SystemDatasetInstantiator.java:109) ~[na:na]
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$1.load(SingleThreadDatasetCache.java:98) ~[na:na]
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$1.load(SingleThreadDatasetCache.java:94) ~[na:na]
              at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache.get(LocalCache.java:3990) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) ~[guava-13.0.1.jar:na]
              at com.google.common.cache.ForwardingLoadingCache.get(ForwardingLoadingCache.java:48) ~[guava-13.0.1.jar:na]
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$LineageRecordingDatasetCache.get(SingleThreadDatasetCache.java:151) ~[na:na]
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$LineageRecordingDatasetCache.get(SingleThreadDatasetCache.java:134) ~[na:na]
              at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache.getDataset(SingleThreadDatasetCache.java:177) ~[na:na]
              at co.cask.cdap.data2.dataset2.MultiThreadDatasetCache.getDataset(MultiThreadDatasetCache.java:114) ~[na:na]
              at co.cask.cdap.data2.dataset2.DynamicDatasetCache.getDataset(DynamicDatasetCache.java:203) ~[na:na]
              at co.cask.cdap.data2.dataset2.DynamicDatasetCache.getDataset(DynamicDatasetCache.java:162) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkTransactional$TransactionalDatasetContext.getDataset(SparkTransactional.java:368) ~[na:na]
              at co.cask.cdap.app.runtime.spark.AbstractSparkExecutionContext$$anon$7.run(AbstractSparkExecutionContext.scala:439) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:206) ~[na:na]
              at co.cask.cdap.app.runtime.spark.AbstractSparkExecutionContext.saveAsDataset(AbstractSparkExecutionContext.scala:437) ~[na:na]
              at co.cask.cdap.app.runtime.spark.AbstractSparkExecutionContext.saveAsDataset(AbstractSparkExecutionContext.scala:343) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SerializableSparkExecutionContext.saveAsDataset(SerializableSparkExecutionContext.scala:70) ~[na:na]
              at co.cask.cdap.app.runtime.spark.DefaultJavaSparkExecutionContext.saveAsDataset(DefaultJavaSparkExecutionContext.scala:210) ~[na:na]
              at co.cask.cdap.app.runtime.spark.DefaultJavaSparkExecutionContext.saveAsDataset(DefaultJavaSparkExecutionContext.scala:202) ~[na:na]
              at co.cask.cdap.etl.spark.batch.SparkBatchSinkFactory.writeFromRDD(SparkBatchSinkFactory.java:103) ~[hydrator-spark-core-4.3.0.jar:na]
              at co.cask.cdap.etl.spark.batch.RDDCollection.store(RDDCollection.java:160) ~[hydrator-spark-core-4.3.0.jar:na]
              at co.cask.cdap.etl.spark.SparkPipelineRunner.runPipeline(SparkPipelineRunner.java:181) ~[hydrator-spark-core-4.3.0.jar:na]
              at co.cask.cdap.etl.spark.batch.BatchSparkPipelineDriver.run(BatchSparkPipelineDriver.java:151) ~[hydrator-spark-core-4.3.0.jar:na]
              at co.cask.cdap.api.Transactionals$2.run(Transactionals.java:72) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkTransactional$2.run(SparkTransactional.java:234) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:206) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:136) ~[na:na]
              at co.cask.cdap.app.runtime.spark.AbstractSparkExecutionContext.execute(AbstractSparkExecutionContext.scala:206) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SerializableSparkExecutionContext.execute(SerializableSparkExecutionContext.scala:62) ~[na:na]
              at co.cask.cdap.app.runtime.spark.DefaultJavaSparkExecutionContext.execute(DefaultJavaSparkExecutionContext.scala:96) ~[na:na]
              at co.cask.cdap.api.Transactionals.execute(Transactionals.java:69) ~[na:na]
              at co.cask.cdap.etl.spark.batch.BatchSparkPipelineDriver.run(BatchSparkPipelineDriver.java:118) ~[hydrator-spark-core-4.3.0.jar:na]
              at co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:82) ~[na:na]
              at co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala) ~[na:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552) ~[spark-assembly.jar:1.6.0-cdh5.11.0]
      Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
              at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) ~[hbase-client-1.2.0.jar:1.2.0]
              at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:433) ~[hbase-client-1.2.0.jar:1.2.0]
              at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:426) ~[hbase-client-1.2.0.jar:1.2.0]
              at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:304) ~[hbase-client-1.2.0.jar:1.2.0]
              at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:185) ~[hbase-client-1.2.0.jar:1.2.0]
              at co.cask.cdap.data2.util.hbase.HBase12CDH570TableUtil.createHTable(HBase12CDH570TableUtil.java:63) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.HBaseTable.<init>(HBaseTable.java:113) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.HBaseTableDefinition.getDataset(HBaseTableDefinition.java:50) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.table.hbase.HBaseTableDefinition.getDataset(HBaseTableDefinition.java:34) ~[na:na]
              at co.cask.cdap.api.dataset.lib.CompositeDatasetDefinition.getDataset(CompositeDatasetDefinition.java:114) ~[na:na]
              at co.cask.cdap.api.dataset.lib.IndexedTableDefinition.getDataset(IndexedTableDefinition.java:86) ~[na:na]
              at co.cask.cdap.api.dataset.lib.IndexedTableDefinition.getDataset(IndexedTableDefinition.java:37) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetDefinition.getDataset(TimePartitionedFileSetDefinition.java:86) ~[na:na]
              at co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetDefinition.getDataset(TimePartitionedFileSetDefinition.java:45) ~[na:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetType.getDataset(DatasetType.java:73) ~[na:na]
              at co.cask.cdap.data2.datafabric.dataset.RemoteDatasetFramework.getDataset(RemoteDatasetFramework.java:258) ~[na:na]
              at co.cask.cdap.data2.dataset2.ForwardingDatasetFramework.getDataset(ForwardingDatasetFramework.java:172) ~[na:na]
              at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework.access$101(LineageWriterDatasetFramework.java:63) ~[na:na]
              at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework$2.call(LineageWriterDatasetFramework.java:180) ~[na:na]
              at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework$2.call(LineageWriterDatasetFramework.java:177) ~[na:na]
              at co.cask.cdap.data2.dataset2.DefaultDatasetRuntimeContext.execute(DefaultDatasetRuntimeContext.java:120) ~[na:na]
              at co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework.getDataset(LineageWriterDatasetFramework.java:175) ~[na:na]
              at co.cask.cdap.data2.dataset2.ForwardingDatasetFramework.getDataset(ForwardingDatasetFramework.java:172) ~[na:na]
              at co.cask.cdap.internal.app.runtime.workflow.NameMappedDatasetFramework.getDataset(NameMappedDatasetFramework.java:156) ~[na:na]
              at co.cask.cdap.data.dataset.SystemDatasetInstantiator.getDataset(SystemDatasetInstantiator.java:101) ~[na:na]
              ... 44 common frames omitted
      Caused by: java.lang.reflect.InvocationTargetException: null
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_75]
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_75]
              at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ~[hbase-client-1.2.0.jar:1.2.0]
              ... 68 common frames omitted
      Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
              at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2231) ~[spark-assembly.jar:1.6.0-cdh5.11.0]
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:665) ~[hbase-client-1.2.0.jar:1.2.0]
              ... 73 common frames omitted
      Caused by: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
              at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2225) ~[spark-assembly.jar:1.6.0-cdh5.11.0]
              ... 74 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: