Many of our examples define the programs of an app as inner classes of the app. This works find in most cases. However, it does not work if one the classes is a Spark program. For example, if the app contains a workflow custom action and a Spark program as inner classes, then the custom action will fail with exception:
Apparently, this is inherent to how the JVM loads inner classes. Because the custom action is an inner class of a class that also has an inner class that depends on spark, the custom action also depends on spark. But during execution of the custom action, because it is not a Spark program, we do not make Spark available to the class loader.
This may also be a problem with other program types.
The workaround for this is to declare the Spark programs as top-level classes.
There are several ways we can address this:
- Somehow figure out whether the custom action depends on Spark and make that available to the workflow class loader. That would also help if a custom action actually depends on Spark.
- Disallow Spark programs that are not at top-level. However, that would be restrictive, because in many cases this does not cause trouble.
- Document clearly in the documentation what the issue is, what causes it, and how to avoid it
- Document it as a known issue in release notes.