Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-9119

The log collection system should warn the user if an app floods the logs

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.1.0
    • Fix Version/s: None
    • Component/s: Log
    • Labels:
    • Rank:
      1|hzzyd3:

      Description

      Applications can sometimes, unintentionally, emit a lot of logs. One example of such a case when a previously well-behaved mapreduce now received data that looks slightly different. The code might issue a warning because it does not expect that data; and that will result in a flood of identical messages, one for every data event.

      The consequence is that the app emits so many messages that the log collection system cannot keep up (kafka log dir is full, log saver falls behind or runs out of memory, etc.). But once the log collection fails, it becomes even harder for the user to find the root cause of the problem - he will assume it is an operational issue with CDAP.

      It would be much better if the log collection would detect such a misbehavior and notifies the user of it. For example,

      • send a notification on TMS?
      • emit en error message?
      • show an alert in the UI

      This will help the user to detect systemic application issues early, and the operator avoid spending time diagnosing the logging system.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bhooshan Bhooshan Mogal
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: