Applications can sometimes, unintentionally, emit a lot of logs. One example of such a case when a previously well-behaved mapreduce now received data that looks slightly different. The code might issue a warning because it does not expect that data; and that will result in a flood of identical messages, one for every data event.
The consequence is that the app emits so many messages that the log collection system cannot keep up (kafka log dir is full, log saver falls behind or runs out of memory, etc.). But once the log collection fails, it becomes even harder for the user to find the root cause of the problem - he will assume it is an operational issue with CDAP.
It would be much better if the log collection would detect such a misbehavior and notifies the user of it. For example,
- send a notification on TMS?
- emit en error message?
- show an alert in the UI
This will help the user to detect systemic application issues early, and the operator avoid spending time diagnosing the logging system.