Sometimes metrics for the spark streaming does not get show up in the UI on pipeline page, but these metrics can be seen using dashboard.
Issue happens because we first query for the all available metric names (user and system) for a particular program. The list is then filter to get the user metric names. Current filtering logic
The workaround is to label the node differently so that "contains" check fail for the system metrics.