Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13919

Reporting spark application doesn't work on spark 2.0 - fail gracefully.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.0
    • Fix Version/s: 5.0.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|i00fev:

      Description

      While testing on CM clusters, we noticed that the reporting application works well with spark 2.1 and spark 2.2, However it fails on spark 2.0 due to the issue with spark-avro library we use to read/write avro files in spark.

      we use 4.0 version of spark-avro, however 3.2 version is suggested spark-avro version for spark 2.0. 

      we cannot use 3.2 for version for spark-avro as default as it doesn't work with spark 2.2

      Since the spark2 - 2.1 version is the default version used in sdk and cdap-spark-core, its not clear if all of our spark applications would work on spark 2.0, however we need to handle reporting application to fail/error gracefully on spark-2.0 if its decided that reporting will not be supported for versions lower than spark 2.1

        Attachments

          Activity

            People

            • Assignee:
              shankar Shankar Selvam
              Reporter:
              shankar Shankar Selvam
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: