Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-1905

Explore broken on secure cdh 5.x clusters

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.8.0
    • Fix Version/s: 3.1.0
    • Component/s: Explore
    • Labels:
      None
    • Rank:
      1|hzypmn:

      Description

      The explore service is unable to start on a secure 5.x cluster.

      On cdh 5.2 and 5.3, we see the following exception:

      2015-03-21 01:31:50,059 - ERROR [ExploreExecutorService STARTING:c.c.c.c.t.AbstractMasterTwillRunnable$1@144] - Service co.cask.cdap.explore.executor.ExploreExecutorService failed
      com.google.common.util.concurrent.UncheckedExecutionException: org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
              at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractService.startAndWait(AbstractService.java:220) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractIdleService.startAndWait(AbstractIdleService.java:106) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.explore.executor.ExploreExecutorService.startUp(ExploreExecutorService.java:90) ~[co.cask.cdap.cdap-explore-2.8.0-SNAPSHOT.jar:na]
              at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) ~[com.google.guava.guava-13.0.1.jar:na]
              at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_34]
      Caused by: org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
              at org.apache.hive.service.cli.CLIService.init(CLIService.java:88) ~[hive-service-0.13.1-cdh5.3.2.jar:0.13.1-cdh5.3.2]
              at co.cask.cdap.explore.service.hive.BaseHiveExploreService.startUp(BaseHiveExploreService.java:275) ~[co.cask.cdap.cdap-explore-2.8.0-SNAPSHOT.jar:na]
              ... 2 common frames omitted
      Caused by: java.io.IOException: HiveServer2 kerberos principal or keytab is not correctly configured
              at org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:183) ~[hive-service-0.13.1-cdh5.3.2.jar:0.13.1-cdh5.3.2]
              at org.apache.hive.service.cli.CLIService.init(CLIService.java:85) ~[hive-service-0.13.1-cdh5.3.2.jar:0.13.1-cdh5.3.2]
              ... 3 common frames omitted
      

      with cdh 5.0, the service starts up but queries dont go through. For example, a 'show tables' query results in the following exception:

      2015-03-21 02:22:21,423 - ERROR [pool-27-thread-1:o.a.t.t.TSaslTransport@296] - SASL negotiation failure
      javax.security.sasl.SaslException: GSS initiate failed
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[na:1.6.0_34]
      	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0]
      	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0]
      	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0]
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_34]
      	at javax.security.auth.Subject.doAs(Subject.java:416) ~[na:1.6.0_34]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[hadoop-common-2.3.0-cdh5.0.0.jar:na]
      	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:288) ~[hive-metastore-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169) ~[hive-metastore-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_34]
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.6.0_34]
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.6.0_34]
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:534) ~[na:1.6.0_34]
      	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1161) ~[hive-metastore-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) ~[hive-metastore-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) ~[hive-metastore-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2407) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2418) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1141) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1130) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2250) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:334) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:926) ~[hive-exec-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:144) ~[hive-service-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:64) ~[hive-service-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:177) ~[hive-service-0.12.0-cdh5.0.0.jar:0.12.0-cdh5.0.0]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_34]
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_34]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_34]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_34]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_34]
      	at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_34]
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.6.0_34]
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.6.0_34]
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.6.0_34]
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:218) ~[na:1.6.0_34]
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:213) ~[na:1.6.0_34]
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180) ~[na:1.6.0_34]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[na:1.6.0_34]
      	... 40 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                alvin Alvin Wang
                Reporter:
                ashau Albert Shau
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: