Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-8295

Unable to create namespaces on a cluster without Kerberos enabled

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.1.0
    • Fix Version/s: 4.1.0
    • Component/s: Namespaces, Security
    • Labels:
      None
    • Rank:
      1|hzztrb:

      Description

      The following errors happen because on a non-Kerberos cluster, HDFS operations performed by the cdap user occur as the 'hdfs.user', which can be something other than 'cdap'. For instance, the files and directories created by CDAP can be owned by the 'yarn' user, if 'hdfs.user' is set to 'yarn'.

      [ali@<HOSTNAME> ~]$ hdfs dfs -ls /cdap/namespaces
      Found 1 items
      drwxr-xr-x   - yarn supergroup          0 2017-01-31 10:39 /cdap/namespaces/system
      [ali@<HOSTNAME> ~]$ hdfs dfs -ls /cdap
      Found 5 items
      drwxr-xr-x   - yarn supergroup          0 2017-01-31 10:39 /cdap/cdap
      drwxr-xr-x   - yarn supergroup          0 2017-01-31 23:10 /cdap/namespaces
      drwxrwxrwx   - yarn supergroup          0 2017-01-31 10:39 /cdap/tms
      drwxrwxrwx   - yarn supergroup          0 2017-01-31 10:40 /cdap/twill
      drwxr-xr-x   - yarn supergroup          0 2017-01-31 23:10 /cdap/tx.snapshot
      

      From master logs:

      2017-01-31 23:07:05,996 - DEBUG [Endure-Service-:c.c.c.c.s.RetryOnStartFailureService$1@70] - Exception raised when starting service 
      java.util.concurrent.ExecutionException: co.cask.cdap.common.NamespaceCannotBeCreatedException: 'namespace:default' cannot be created. Reason: User does not belong to cdap
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:808)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
      
              at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na]
              at co.cask.cdap.common.service.RetryOnStartFailureService$1.run(RetryOnStartFailureService.java:63) ~[na:na]
      Caused by: co.cask.cdap.common.NamespaceCannotBeCreatedException: 'namespace:default' cannot be created. Reason: User does not belong to cdap
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:808)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
      
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.create(DefaultNamespaceAdmin.java:194) ~[na:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceEnsurer$1$1.doStart(DefaultNamespaceEnsurer.java:51) ~[na:na]
              at com.google.common.util.concurrent.AbstractService.start(AbstractService.java:170) ~[com.google.guava.guava-13.0.1.jar:na]
              ... 1 common frames omitted
      Caused by: org.apache.hadoop.security.AccessControlException: User does not belong to cdap
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:808)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_75]
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_75]
              at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.hdfs.DFSClient.setOwner(DFSClient.java:2524) ~[hadoop-hdfs-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.fs.Hdfs.setOwner(Hdfs.java:342) ~[hadoop-hdfs-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.fs.FileContext$12.next(FileContext.java:1045) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.fs.FileContext$12.next(FileContext.java:1041) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.fs.FileContext.setOwner(FileContext.java:1041) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.twill.filesystem.FileContextLocation.setGroup(FileContextLocation.java:108) ~[org.apache.twill.twill-yarn-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
              at co.cask.cdap.internal.app.namespace.AbstractStorageProviderNamespaceAdmin.createLocation(AbstractStorageProviderNamespaceAdmin.java:156) ~[na:na]
              at co.cask.cdap.internal.app.namespace.AbstractStorageProviderNamespaceAdmin.create(AbstractStorageProviderNamespaceAdmin.java:72) ~[na:na]
              at co.cask.cdap.internal.app.namespace.DistributedStorageProviderNamespaceAdmin.create(DistributedStorageProviderNamespaceAdmin.java:68) ~[na:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin$2.call(DefaultNamespaceAdmin.java:186) ~[na:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin$2.call(DefaultNamespaceAdmin.java:183) ~[na:na]
              at co.cask.cdap.common.security.ImpersonationUtils$1.run(ImpersonationUtils.java:46) ~[na:na]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
              at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_75]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at co.cask.cdap.common.security.ImpersonationUtils.doAs(ImpersonationUtils.java:43) ~[na:na]
              at co.cask.cdap.common.security.DefaultImpersonator.doAs(DefaultImpersonator.java:65) ~[na:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.create(DefaultNamespaceAdmin.java:183) ~[na:na]
              ... 3 common frames omitted
      Caused by: org.apache.hadoop.ipc.RemoteException: User does not belong to cdap
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:808)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:1455) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.ipc.Client.call(Client.java:1392) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at com.sun.proxy.$Proxy22.setOwner(Unknown Source) ~[na:na]
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setOwner(ClientNamenodeProtocolTranslatorPB.java:383) ~[hadoop-hdfs-2.7.1.2.3.6.0-3796.jar:na]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
              at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) ~[hadoop-common-2.7.1.2.3.6.0-3796.jar:na]
              at com.sun.proxy.$Proxy23.setOwner(Unknown Source) ~[na:na]
              at org.apache.hadoop.hdfs.DFSClient.setOwner(DFSClient.java:2522) ~[hadoop-hdfs-2.7.1.2.3.6.0-3796.jar:na]
              ... 21 common frames omitted
      

      The same error happens upon attempting to create any namespace:

      cdap (host.net:11015/namespace:default)> create namespace foo
      Error: 500: User does not belong to cdap
      	at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:808)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: