When the meta store is restarted, it loses all delegation tokens that were previously acquired, and all clients must obtain new delegation tokens. However, in CDAP explore.service, no new token is acquired. Instead, we see, an exception in the logs:
and subsequently, all queries fail to ever produce results and remain in RUNNING state. The hive client will periodically log that it could not connect to the meta store and sleep for 5 seconds, then repeat. But all queries remain in running state forever.
While that is a HIve issue (queries should actually fail), CDAP should have a way to recognize that its delegation token has expired and request a new one.
Currently this requires a restart of CDAP - actually a shutdown and start, to make sure the master.services app is restarted with new tokens.
If this cannot happen automatically, there should at least be a way to surface the problem (show explore.service as not healthy in UI?) and have a way to manually trigger renewal of the delegation token,