Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-1865

Explore cannot connect to secure Hive metastore

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 2.8.0
    • Component/s: Explore
    • Labels:
      None
    • Rank:
      1|hzypev:

      Description

      With explore enabled on a secure cluster, I get an exception whenever the explore service tries to talk to the Hive metastore.

      hive-site.xml has 3 settings added to enable auth

          <property>
            <name>hive.metastore.sasl.enabled</name>
            <value>true</value>
          </property>
          <property>
            <name>hive.metastore.kerberos.keytab.file</name>
            <value>/etc/security/keytabs/hive.service.keytab</value>
          </property>
          <property>
            <name>hive.metastore.kerberos.principal</name>
            <value>hive/_HOST@{domain here}</value>
          </property>
      

      Then, when explore service starts up I see an exception when it tries to get databases.

      Exception in thread "Thread-5" com.google.common.util.concurrent.UncheckedExecutionException: java.lang.Exception: Service failed while running
      	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015)
      	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
      	at com.google.common.util.concurrent.AbstractService.stopAndWait(AbstractService.java:225)
      	at com.google.common.util.concurrent.AbstractExecutionThreadService.stopAndWait(AbstractExecutionThreadService.java:170)
      	at org.apache.twill.internal.ServiceMain$1.run(ServiceMain.java:82)
      Caused by: java.lang.Exception: Service failed while running
      	at com.google.common.util.concurrent.AbstractService$1.failed(AbstractService.java:123)
      	at com.google.common.util.concurrent.AbstractService$6$1.run(AbstractService.java:444)
      	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
      	at com.google.common.util.concurrent.AbstractService$ListenerExecutorPair.execute(AbstractService.java:470)
      	at com.google.common.util.concurrent.AbstractService$6.run(AbstractService.java:442)
      	at com.google.common.util.concurrent.AbstractService.executeListeners(AbstractService.java:369)
      	at com.google.common.util.concurrent.AbstractService.notifyFailed(AbstractService.java:313)
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:68)
      	at java.lang.Thread.run(Thread.java:744)
      Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
      	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015)
      	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
      	at com.google.common.util.concurrent.AbstractService.startAndWait(AbstractService.java:220)
      	at com.google.common.util.concurrent.AbstractIdleService.startAndWait(AbstractIdleService.java:106)
      	at co.cask.cdap.explore.executor.ExploreExecutorService.startUp(ExploreExecutorService.java:89)
      	at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
      	... 1 more
      Caused by: org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
      	at org.apache.hive.service.cli.CLIService.start(CLIService.java:139)
      	at co.cask.cdap.explore.service.hive.BaseHiveExploreService.startUp(BaseHiveExploreService.java:276)
      	... 2 more
      Caused by: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDODataStoreException: Communications link failure
      
      The last packet successfully received from the server was 564,792 milliseconds ago.  The last packet sent successfully to the server was 0 milliseconds ago.
      NestedThrowables:
      com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
      
      The last packet successfully received from the server was 564,792 milliseconds ago.  The last packet sent successfully to the server was 0 milliseconds ago.)
      	at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1112)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:828)
      	at org.apache.hive.service.cli.CLIService.start(CLIService.java:137)
      	... 3 more
      

      Seems like the explore service is not kerberos authenticated.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ashau Albert Shau
                Reporter:
                ashau Albert Shau
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: