Solved ERR 20: Auth Credentials are too weak (NFSv4 + KRB5)

monwarez · Dec 5, 2022

I have

Code:

nfsuserd_flags="-manage-gids -domain NFSDOMAIN"

In my /etc/rc.conf, this seems to be for AUTH_SYS, so I don't really think that it will solve it.

spmzt · Dec 5, 2022

monwarez said:
I have

Code:

nfsuserd_flags="-manage-gids -domain NFSDOMAIN"

In my /etc/rc.conf, this seems to be for AUTH_SYS, so I don't really think that it will solve it.

I read it before in the manual. The domain flag should not be necessary since it can be read from the server's hostname. However I will test the manage-gids + domain flags soon.

spmzt · Dec 5, 2022

covacat said:
i got that too
the problem is that your client can't traverse server mountpoints
export every volume separately

For sake of simplicity, I only have one dataset on the NFS root and it's exported already

covacat · Dec 5, 2022

here is what i have

Code:

# kdc, nfs server
root@linsux:~ # kadmin -l
kadmin> list -s *
Principal                Expiration  PW-exp  PW-change   Max life   Max renew
covacat                    never       never   2022-12-05  1 day      1 week
default                  never       never   2022-12-05  1 day      1 week
kadmin/admin             never       never   2022-12-05  1 hour     1 hour
kadmin/hprop             never       never   2022-12-05  1 hour     1 hour
kadmin/changepw          never       never   2022-12-05  5 minutes  5 minutes
changepw/kerberos        never       never   2022-12-05  1 hour     1 hour
krbtgt/MYDOMAIN.LOCAL      never       never   2022-12-05  unlimited  unlimited
WELLKNOWN/ANONYMOUS      never       never   2022-12-05  1 hour     1 hour
nfs/linsux.mydomain.local  never       never   2022-12-05  1 day      1 week
host/hpbsd.mydomain.local  never       never   2022-12-05  1 day      1 week

linsux:~ # tail -7 /etc/rc.conf
kerberos5_server_enable="YES"
kadmind5_server_enable="YES"
nfsuserd_enable="YES"
gssd_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
mountd_enable="YES"

root@linsux:~ # cat /etc/exports
/extra/wwwroot -sec=krb5p -network=10.1.1.0/24
V4: /extra/wwwroot -sec=krb5p -network=10.1.1.0/24

#----- client -----
[covacat@hpbsd ~]$ tail -4 /etc/rc.conf
nfsuserd_enable="YES"
gssd_enable="YES"
nfs_client_enable="YES"
nfscbd_enable="YES"

# mount command

mount -t nfs -o nfsv4,sec=krb5p,gssname=host,intr linsux.mydomain.local:/ /extra

spmzt · Dec 6, 2022

spmzt said:
Somehow I make it work.
with these configuration:
/etc/rc.conf:
... gssd_enable="YES" gssd_flags="-h" nfsuserd_enable="YES" nfs_server_enable="YES" nfsv4_server_only="YES" nfs_server_flags="-t -n 32"

/etc/exports:
V4: /tank/ds -sec=krb5p -network *.*.*.0/24

And:
# zfs get sharenfs tank/ds NAME PROPERTY VALUE SOURCE tank/ds sharenfs sec=krb5p local

Plus previous configuration on the client.
But my nfs client works with *nobody* permission.
# touch /storage/ds/ds0/test2 root@client:~ # ls -ltrha /storage/ds/ds0 total 20 drwxr-xr-x 22 root wheel 22B Dec 3 21:02 .. -r--r--r--+ 1 nobody wheel 0B Dec 5 21:01 test

I Tried a normal user and got a ticket for it.
Hopefully, It works now, For FreeBSD Client.

spmzt · Dec 6, 2022

Now I have the same "ERR 20: Auth Credentials are too weak" on my linux client (ubuntu).

client-side:
rpc.gssd running in foreground mode:

 # rpc.gssd -f -vvv -r

libtirpc: debug level 1



handle_gssd_upcall(0x7fc697f22740): 'mech=krb5 uid=0 service=* enctypes=18,17,16' (nfs/clnt1c)

start_upcall_thread(0x7fc697f22740): created thread id 0x7fc697720640

krb5_use_machine_creds(0x7fc697720640): uid 0 tgtname (null)

No key table entry found for ubuntu-client$@DOMAIN.NET while getting keytab entry for 'ubuntu-client$@DOMAIN.NET'

No key table entry found for ubuntu-client$@DOMAIN.NET while getting keytab entry for 'ubuntu-client$@DOMAIN.NET'

No key table entry found for root/ubuntu-client.DOMAIN.NET@DOMAIN.NET while getting keytab entry for 'root/ubuntu-client.DOMAIN.NET@DOMAIN.NET'

find_keytab_entry(0x7fc697720640): Success getting keytab entry for 'nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET'

gssd_get_single_krb5_cred(0x7fc697720640): principal 'nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET' ccache:'FILE:/tmp/krb5ccmachine_DOMAIN.NET'

gssd_get_single_krb5_cred(0x7fc697720640): Credentials in CC 'FILE:/tmp/krb5ccmachine_DOMAIN.NET' are good until Wed Dec  7 02:47:01 2022

create_auth_rpc_client(0x7fc697720640): creating tcp client for server server.DOMAIN.NET

create_auth_rpc_client(0x7fc697720640): creating context with server nfs@server.DOMAIN.NET

do_downcall(0x7fc697720640): lifetime_rec=24h:0m:0s acceptor=nfs@server.DOMAIN.NET

WARNING: handle_gssd_upcall: failed reading request

Then:

 # mount -t nfs -o vers=4.2,tcp,sec=krb5p,intr server.DOMAIN.NET:/ /mnt -vvvv

mount.nfs: timeout set for Tue Dec  6 02:50:38 2022

mount.nfs: trying text-based options 'vers=4.2,tcp,sec=krb5p,intr,addr=*.*.*.*,clientaddr=*.*.*.*'

mount.nfs: mount(2): Permission denied

mount.nfs: access denied by server while mounting server.DOMAIN.NET:/

klist:

 # klist -v

Credentials cache: FILE:/tmp/krb5cc_0

        Principal: nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET

    Cache version: 4



Server: krbtgt/DOMAIN.NET@DOMAIN.NET

Client: nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET

Ticket etype: aes256-cts-hmac-sha1-96, kvno 1

Ticket length: 344

Auth time:  Dec  6 02:57:50 2022

End time:   Dec  7 02:57:50 2022

Ticket flags: pre-authent, initial, forwardable

Addresses: addressless

Meanwhile in nfs server:

 # tail -f /var/log/daemon.log

gssd[6868]: gssd_accept_sec_context: done major=0x0 minor=0

gssd[6868]: gssd_export_sec_context: done major=0x0 minor=0

gssd[6868]: gssd_export_name: done major=0x0 minor=0

gssd[6868]: gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227

gssd[6868]: gssd_release_name: done major=0x0 minor=0

gssd[6868]: gssd_release_cred: done major=0x0 minor=0

gssd[6868]: gssd_accept_sec_context: done major=0x0 minor=0

gssd[6868]: gssd_export_sec_context: done major=0x0 minor=0

gssd[6868]: gssd_export_name: done major=0x0 minor=0

gssd[6868]: gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227

gssd[6868]: gssd_release_name: done major=0x0 minor=0

gssd[6868]: gssd_release_cred: done major=0x0 minor=0

Any Ideas?

spmzt · Dec 6, 2022

spmzt said:
Now I have the same "ERR 20: Auth Credentials are too weak" on my linux client (ubuntu).

client-side:
rpc.gssd running in foreground mode:
# rpc.gssd -f -vvv -r libtirpc: debug level 1 handle_gssd_upcall(0x7fc697f22740): 'mech=krb5 uid=0 service=* enctypes=18,17,16' (nfs/clnt1c) start_upcall_thread(0x7fc697f22740): created thread id 0x7fc697720640 krb5_use_machine_creds(0x7fc697720640): uid 0 tgtname (null) No key table entry found for ubuntu-client$@DOMAIN.NET while getting keytab entry for 'ubuntu-client$@DOMAIN.NET' No key table entry found for ubuntu-client$@DOMAIN.NET while getting keytab entry for 'ubuntu-client$@DOMAIN.NET' No key table entry found for root/ubuntu-client.DOMAIN.NET@DOMAIN.NET while getting keytab entry for 'root/ubuntu-client.DOMAIN.NET@DOMAIN.NET' find_keytab_entry(0x7fc697720640): Success getting keytab entry for 'nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET' gssd_get_single_krb5_cred(0x7fc697720640): principal 'nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET' ccache:'FILE:/tmp/krb5ccmachine_DOMAIN.NET' gssd_get_single_krb5_cred(0x7fc697720640): Credentials in CC 'FILE:/tmp/krb5ccmachine_DOMAIN.NET' are good until Wed Dec 7 02:47:01 2022 create_auth_rpc_client(0x7fc697720640): creating tcp client for server server.DOMAIN.NET create_auth_rpc_client(0x7fc697720640): creating context with server nfs@server.DOMAIN.NET do_downcall(0x7fc697720640): lifetime_rec=24h:0m:0s acceptor=nfs@server.DOMAIN.NET WARNING: handle_gssd_upcall: failed reading request

Then:
# mount -t nfs -o vers=4.2,tcp,sec=krb5p,intr server.DOMAIN.NET:/ /mnt -vvvv mount.nfs: timeout set for Tue Dec 6 02:50:38 2022 mount.nfs: trying text-based options 'vers=4.2,tcp,sec=krb5p,intr,addr=*.*.*.*,clientaddr=*.*.*.*' mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting server.DOMAIN.NET:/

klist:
# klist -v Credentials cache: FILE:/tmp/krb5cc_0 Principal: nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET Cache version: 4 Server: krbtgt/DOMAIN.NET@DOMAIN.NET Client: nfs/ubuntu-client.DOMAIN.NET@DOMAIN.NET Ticket etype: aes256-cts-hmac-sha1-96, kvno 1 Ticket length: 344 Auth time: Dec 6 02:57:50 2022 End time: Dec 7 02:57:50 2022 Ticket flags: pre-authent, initial, forwardable Addresses: addressless

Meanwhile in nfs server:
# tail -f /var/log/daemon.log gssd[6868]: gssd_accept_sec_context: done major=0x0 minor=0 gssd[6868]: gssd_export_sec_context: done major=0x0 minor=0 gssd[6868]: gssd_export_name: done major=0x0 minor=0 gssd[6868]: gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227 gssd[6868]: gssd_release_name: done major=0x0 minor=0 gssd[6868]: gssd_release_cred: done major=0x0 minor=0 gssd[6868]: gssd_accept_sec_context: done major=0x0 minor=0 gssd[6868]: gssd_export_sec_context: done major=0x0 minor=0 gssd[6868]: gssd_export_name: done major=0x0 minor=0 gssd[6868]: gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227 gssd[6868]: gssd_release_name: done major=0x0 minor=0 gssd[6868]: gssd_release_cred: done major=0x0 minor=0

Any Ideas?

Found this:

Mounting FreeBSD NFSv4 share on Linux using krb5

groups.google.com

It seems like we can not use krb5p on linux nfs implementation.

For future references, Here is what I have done:

 root@server:~# cat /etc/exports

V4: /tank/ds -sec=krb5i:krb5p -network *.*.*.0/24

root@server:~ # zfs get sharenfs

NAME                                       PROPERTY  VALUE            SOURCE

tank/ds                                    sharenfs  sec=krb5p:krb5i  local

xk2600 · Oct 24, 2023

Thank you to spmzt and the many others that have shared their configurations here.. I have been trying to get kerberized NFS working again today after a failed ZFS cluster left me building a cluster from scratch. (My customer made some poor decisions to avoid calling support.) Most of the data was recovered, but I hadn't backed up the actual root of the NFS server so I didn't have the /etc configurations handy.

I want to point out my folly in case others run into this same issue. First configs for the tl;dr folks..

sh:

# NFS Server -- /etc/rc.conf
hostname="storage-1.domain.local"
zfs_enable="YES"
ntpd_enable="YES"
ntpdate_enable="YES"

gssd_enable="YES"
# log GSSD to /var/log/daemon.log
gssd_flags="-h -v"
nfs_server_enable="YES"
nfs_server_flags="-t -n 32"
nfsuserd_enable="YES"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"

NOTE: Fully-quallified domain-name for the hostname, so gssd asks for the correct ticket.

sh:

# NFS Server -- /etc/exports
V4: /srv

NOTE: I am only using the /etc/exports to set the root for the NFSv4 server. All actual security will be set with the zfs set sharenfs attribute and show up in /etc/zfs/exports.

sh:

# zfs get -r -t filesystem sharenfs zstorage
NAME                                            PROPERTY  VALUE            SOURCE
zstorage                                        sharenfs  off              default
zstorage/recovered                              sharenfs  off              default
zstorage/srv                                    sharenfs  sec=krb5i:krb5p  local
zstorage/srv/backup                             sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/backup/rhino                       sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/backup/warden                      sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/eve                                sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/home                               sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img                                sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/bhyve                          sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/firmware                       sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/iso                            sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/ova                            sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/sys                            sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/img/virtualbox                     sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/pub                                sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/src                                sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/sys                                sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/sys/FreeBSD                        sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/sys/FreeBSD/13-RELEASE             sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/tftp                               sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/vm                                 sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/vm/backup                          sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv
zstorage/srv/vm/images                          sharenfs  sec=krb5i:krb5p  inherited from zstorage/srv

Results in the following /etc/zfs/exports file.

sh:

# NFS Server -- /etc/zfs/exports
# !!! DO NOT EDIT THIS FILE MANUALLY !!!
/srv    -sec=krb5i:krb5p
/srv/backup     -sec=krb5i:krb5p
/srv/backup/rhino       -sec=krb5i:krb5p
/srv/backup/warden      -sec=krb5i:krb5p
/srv/eve        -sec=krb5i:krb5p
/srv/home       -sec=krb5i:krb5p
/srv/img        -sec=krb5i:krb5p
/srv/img/bhyve  -sec=krb5i:krb5p
/srv/img/firmware       -sec=krb5i:krb5p
/srv/img/iso    -sec=krb5i:krb5p
/srv/img/ova    -sec=krb5i:krb5p
/srv/img/sys    -sec=krb5i:krb5p
/srv/img/virtualbox     -sec=krb5i:krb5p
/srv/pub        -sec=krb5i:krb5p
/srv/src        -sec=krb5i:krb5p
/srv/sys        -sec=krb5i:krb5p
/srv/sys/FreeBSD        -sec=krb5i:krb5p
/srv/sys/FreeBSD/13-RELEASE     -sec=krb5i:krb5p
/srv/tftp       -sec=krb5i:krb5p
/srv/vm/backup  -sec=krb5i:krb5p
/srv/vm/images  -sec=krb5i:krb5p

After confirming mount /net attached to NFS, I confirmed the NFS session was gss wrapped in wireshark and content was encrypted. I was then able to use zfs set sharenfs sec=krb5i:krb5p -network x.x.x.x/x zstorage/<zfspath> to scope back the networks as appropriate.

Take special care to leave krb5i included in the security flavours or linux clients will not mount. There are over 200 clients so I didn't get a chance to test everything, but most are RHEL/CentOS 7+, or FreeBSD 12+ clients.