Steps to reproduce:
- Go to dataprep
- Create an S3 connection
- Navigate to a folder that potentially has 100s or 1000s of folders or files underneath.
- The UI shows the spinning icon forever
- The backend accumulates a lot of logs. (The logs were rolled over to 5+ files within minutes)
- CDAP becomes super slow
- Eventually it gives up and CDAP shutsdown
The above instance mentions only about S3. It could potentially happen with GCS or any other connection which has a lot of data.