Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12973

upgrading distribution doesn't update CDAP's cache of the spark assembly jar

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.2
    • Fix Version/s: 5.0.0
    • Component/s: CDAP, Spark
    • Labels:
    • Release Notes:
      Fixed a bug that prevented Spark jobs from running after CDAP upgrade due to caching of jars.
    • Rank:
      1|i00a1z:

      Description

      When upgrading the the underlying Hadoop distribution, CDAP doesn't reload the spark-assembly.jar. This can cause CDAP spark job to fail with connection refused errors due to NoSuchMethodError:

      Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.ipc.Client.getRpcTimeout(Lorg/apache/hadoop/conf/Configuration;)I
              at org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:360) ~[hadoop-hdfs-2.6.0-cdh5.13.0.jar:na]
              at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:695) ~[hadoop-hdfs-2.6.0-cdh5.13.0.jar:na]
              at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) ~[hadoop-hdfs-2.6.0-cdh5.13.0.jar:na]
              at org.apache.hadoop.fs.Hdfs.<init>(Hdfs.java:90) ~[hadoop-hdfs-2.6.0-cdh5.13.0.jar:1.6.0-cdh5.11.0]
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_152]
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_152]
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_152]
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_152]
              at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134) ~[spark-assembly.jar:1.6.0-cdh5.11.0]
              ... 74 common frames omitted
      

      CDAP caches the spark assembly jar under hdfs:///cdap/frameworks/spark, so if a spark-assembly.jar file is cached there from CDH 5.11, then after upgrading to CDH 5.13, it will attempt to reuse the same, old jar.

      A workaround is to remove the directory hdfs:///cdap/frameworks/spark.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              mattwuenschel Matt Wuenschel
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: