Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-4711

Update WikipediaPipelineApp to work with Spark 1.5 and 1.6

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.4.0
    • Component/s: CDAP Examples, Spark
    • Labels:
      None
    • Rank:
      1|hzz5jz:

      Description

      WikipediaPipelineApp does not work on CDH 5.5 or CDH 5.6, which have Spark 1.5.

      java.lang.NoSuchMethodError: org.apache.spark.mllib.clustering.LDA.run(Lorg/apache/spark/rdd/RDD;)Lorg/apache/spark/mllib/clustering/DistributedLDAModel;
      

      This is because the change in the parameter type from DistributedLDAModel to LDAModel:
      https://github.com/cloudera/spark/blob/cdh5.4.9-release/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala#L232
      https://github.com/cloudera/spark/blob/cdh5.5.1-release/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala#L326

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bhooshan Bhooshan Mogal
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: