After CDAP 2.8.1 release, we added at least two expensive operations to getDataset call in DatasetService -
- Check whether namespace exists. This makes a remote call to app-fabric service, which in turn makes a call to Dataset Service to fetch app meta table.
- Register usage. This again makes a remote call to DatasetService to fetch usage dataset.
These are causing every getDataset call to become at least 3 getDataset calls, and causing timeouts.
This also makes a getDataset call depend on another getDataset call, thus bringing in dependency between Dataset Service executor threads. This can lead to executor threads waiting for a call that can never get executed due to unavailability of executor threads.