Metrics can cause a high load on HBase (especially in an already overloaded cluster). When this happens, it would be good to have the option of disbling metrics until the system quiets down.
Since the load on the metrics system is proportional to the number of running containers, high load is really only generated by application containers (for example, a MR with 1000s of tasks).
The CDAP system only runs a handful of edge nodes and less than a dozen system services. It is desirable to keep system metrics always on. But it is also desirable to disable metrics from applications, especially programs that have many tasks. Therefore we will add an option
- in cdap-site.xml (system wide) to control whether app containers emit metrics
- as a program argument to override the system-wide setting per program run, or permanently for a program by setting a preference.
This will allow disabling/enabling program metrics through a preference without requiring a Master restart.