When Twill launches a YARN job on a secure Hadoop cluster, it creates an initial credentials file with delegation tokens. This file is used during the initial run of the container. Environment variable HADOOP_TOKEN_FILE_LOCATION for the container points to this file.
To refresh credentials, a new credentials file is created on HDFS with new delegation tokens (note that the old tokens are not refreshed). And containers load the new credentials directly from HDFS file. This works fine for all CDAP programs.
However when Explore launches Hive jobs, it still uses HADOOP_TOKEN_FILE_LOCATION as credentials file. This leads to Explore jobs failing after the initial tokens expire. We'll need Explore to use the latest credentials while launching new jobs.