Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14441

UI setting num executors for spark incorrectly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1.0
    • Component/s: UI
    • Labels:
      None
    • Sprint:
      5.1 09/25
    • Rank:
      1|i00ian:

      Description

      When the number of executors is set through the UI, the UI ends up passing the following to the backend:

      {
        ...
        "properties": {
            "system.spark.spark.executor.instances": 1
          },
      }
      

      This is incorrect. Everything in the properties map should be a string. Because of this, the spark pipeline will fail with:

      java.lang.IllegalArgumentException: spark.executor.instances should be int, but was 1.0
      

      Somewhere along the way, the 1 gets changed into a 1.0. I think it's when the app config is serialized but not sure. The backend should handle this situation better, but the correct API is for those to be strings.

        Attachments

          Activity

            People

            • Assignee:
              ajai Ajai Narayan
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: