Tried to preview the attached pipeline in Hydrator. Looks like the preview succeeds and the data is there, but the pipeline never seems to stop running. The same pipeline as a MR job runs in 10 seconds.
When publishing the pipeline as a Spark job, the same thing happens. The pipeline runs, metrics come through like it has finished processing, but the job never stops.