Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-4949

Classes in yarn.application.classpath gets picked up first through PluginClassloader

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0, 3.2.0
    • Fix Version/s: None
    • Component/s: CDAP, ETL
    • Labels:
      None
    • Rank:
      1|hzz6rr:

      Description

      We added some classes to yarn.application.classpath through CM and noticed that when an ETL pipeline runs on such a cluster the classes needed by plugins are picked from the above "yarn.application.classpath" jar rather than the one given to PluginClassloader. This goes against the isolation which plugins needs through PluginClassloader.

      Here is the stacktrace:

      Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of co/cask/cdap/internal/app/runtime/plugin/PluginClassLoader) previously initiated loading for a different type with name "org/apache/avro/Schema"
      	at org.apache.hive.hcatalog.mapreduce.SpecialCases.addSpecialCasesParametersToOutputJobProperties(SpecialCases.java:115) ~[hive-hcatalog-core-1.1.0-cdh5.4.5.jar:1.1.0-cdh5.4.5]
      	at org.apache.hive.hcatalog.mapreduce.FosterStorageHandler.configureOutputJobProperties(FosterStorageHandler.java:173) ~[hive-hcatalog-core-1.1.0-cdh5.4.5.jar:1.1.0-cdh5.4.5]
      	at org.apache.hive.hcatalog.common.HCatUtil.configureOutputStorageHandler(HCatUtil.java:498) ~[hive-hcatalog-core-1.1.0-cdh5.4.5.jar:1.1.0-cdh5.4.5]
      	at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:198) ~[hive-hcatalog-core-1.1.0-cdh5.4.5.jar:1.1.0-cdh5.4.5]
      	at co.cask.plugin.etl.batch.sink.HiveBatchSink$HiveSinkOutputFormatProvider.<init>(HiveBatchSink.java:137) ~[1453894836780-0/:na]
      	at co.cask.plugin.etl.batch.sink.HiveBatchSink.prepareRun(HiveBatchSink.java:95) ~[1453894836780-0/:na]
      	at co.cask.plugin.etl.batch.sink.HiveBatchSink.prepareRun(HiveBatchSink.java:66) ~[1453894836780-0/:na]
      	at co.cask.cdap.etl.batch.ETLMapReduce.beforeSubmit(ETLMapReduce.java:183) ~[1453894825620-0/:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:464) ~[co.cask.cdap.cdap-app-fabric-3.2.3-SNAPSHOT.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2.call(MapReduceRuntimeService.java:459) ~[co.cask.cdap.cdap-app-fabric-3.2.3-SNAPSHOT.jar:na]
      	at co.cask.cdap.data2.transaction.Transactions.execute(Transactions.java:108) ~[co.cask.cdap.cdap-data-fabric-3.2.3-SNAPSHOT.jar:na]
      	... 5 common frames omitted
      

        Attachments

          Activity

            People

            • Assignee:
              nitin Nitin Motgi
              Reporter:
              rsinha Rohit Sinha
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: