We need the ability to do point lookups in Transform plugins. We have that feature for KVTables but this feature will not work in Spark due to closure limitations. Also, note that the dataset could be in a different namespace.
The use case is when a record comes in and we want to lookup a property for a field in that record from a dataset. The workaround in Batch pipelines is to have another source that reads from the dataset and performs a join to enrich the data. For this workaround will not work for realtime streaming pipelines.