Steps to reproduce:
1. Run a program P1 that uses a dataset D1. This will be recorded in the usage dataset.
2. Delete the application.
3. Redeploy the application and rerun step 1. The usage will not be recorded in the usage dataset now.
The cause for this is that there's a cache in DefaultUsageRegistry that is supposed to prevent too many hbase writes, with the idea that we don't need to record in hbase, if its already been recorded before. However, when the app is deleted, the usage is also deleted, so the hbase writes (to usage dataset) would need to happen. However, this cache in DefaultUsageRegistry is not invalidated, so additional hbase writes for that app/dataset pair do not happen.
Upon app delete, we do invalidate a DefaultUsageRegistry's cache, but its a different instance than the one actually recording the usage (which is running in Dataset op executor service).
1. Move the usage recording to master service. This means more traffic on CDAP master service, but we can cache in the usage recording service.
2. Don't have a cache in DefaultUsageRegistry in dataset op executor service. Only have it in the clients of this service (i.e. in program containers).