Currently one always need to write a Program (usually a Flow) to get data from external source (e.g. Kafka) and persist it to dataset so that multiple programs can share and process the same dataset.
As part of ZAP, the platform should have the capability to support common data source for ingesting data into stream. It should be designed with a pluggable architecture in mind so that it's easy to add new source/sink type. Also, it should be possible to define simply data cleansing logic before writing into sink.
|Define a pluggable architecture to support ingest pipe||Resolved|
|Kafka source ingest pipe||Resolved|