There is a race condition that can leave metadata around for a deleted program.
When an app is deployed, system metadata for the app is added synchronously in SystemMetadataWriterStage through direct calls to the MetadataHttpHandler.
Similarly, when an app is deleted, system metadata for the app is deleted synchronously in DeletedProgramHandlerStage through direct calls to the MetadataHttpHandler.
When an app is deployed, a CREATE_ENTITY message is written to TMS. Messages are read asynchronously by MetadataSubscriberService, which calls ProfileMetadataMessageProcessor. Here, it will check if the app exists. If so, it will add profile metadata to all application programs. If not, it will not do anything.
The race condition is:
1. Application is deployed by user, system metadata is written.
2. CREATE_ENTITY message is read by MetadataSubscriberService
3. ProfileMetadataMessageProcessor checks that the app exists and it does
4. App is deleted by user, system metadata is dropped
5. ProfileMetadataMessageProcessor adds profile metadata for the program, creating metadata for an entity that no longer exists.