Solved pam_winbind.so broken after upgrade from 12.1 to 12.2-RELEASE

Hello :)

I have a Samba file server setup on an FreeBSD 12.1 with Kerberos / Winbind authentication from a Windows Server 2019 AD

Samba configuration is shown bellow in code snippet 1.

The problem I am running into is the following - after upgrading from 12.1 to 12.2, following the instructions here - https://www.freebsd.org/releases/12.2R/installation/ ,
mounting and accessing the samba shares with AD users stopped working, mapping of groups and user IDs supposedly does not work, as all the users are getting "Permission/Access denied" errors when trying to connect to the exported samba shares.

Samba services are apparently running OK, the exported shares are visible and browsable.

Bash:
[root@fs1 ~]# service samba_server status
nmbd is running as pid 60898.
smbd is running as pid 60903.
winbindd is running as pid 60908.

The only suspicious errors in the logs I am seeing are entries for sshd failing to load pam_winbind.so:

Bash:
Apr 14 11:21:03 fs1 sshd[75370]: in try_dlopen(): /usr/local/lib/pam_winbind.so: /usr/local/lib/libp11-kit.so.0: Undefined symbol "strerror_l@FBSD_1.6"
Apr 14 11:21:03 fs1 sshd[75370]: in openpam_load_module(): no /usr/local/lib/pam_winbind.so found
Apr 14 11:21:03 fs1 sshd[75370]: fatal: PAM: initialisation failed
Apr 14 11:22:15 fs1 sshd[75382]: in try_dlopen(): /usr/local/lib/pam_winbind.so: /usr/local/lib/libp11-kit.so.0: Undefined symbol "strerror_l@FBSD_1.6"
Apr 14 11:22:15 fs1 sshd[75382]: in openpam_load_module(): no /usr/local/lib/pam_winbind.so found
Apr 14 11:22:15 fs1 sshd[75382]: fatal: PAM: initialisation failed

What are those symbols missing? Is pam_winbind.so broken, and if so how can I repair it? I didn't find any exact instructions on the web, do I need to reinstall some packages or rebuild them?

Any other ideas on how to debug the issue? Any help will be highly appreciated, as I spend almost 3 days now on trying to understand the cause of the issue/

Disclaimer: I am fairly new to FreeBSD, and this is a setup I inherited from the old team, so bear with me please

Thanks in advance!


======================================================


1. /usr/local/etc/samba/smb4.conf
Code:
log file = /var/log/samba4/clients/%m.log
log level = 3 auth:10 winbind:5
max log size = 50

#winbind separator = /
winbind cache time = 10
winbind enum users = Yes
winbind enum groups = Yes
winbind use default domain = Yes
winbind refresh tickets = Yes
winbind expand groups = 8
winbind nss info = rfc2307 sfu template

template homedir = /storage/%D/%U
template shell = /usr/local/bin/bash

idmap config *: backend = tdb
idmap config *: range = 90000001-100000000

idmap config ADP:default = Yes
idmap config ADP:backend = rid
idmap config ADP:range = 10000-90000000

load printers = No
printing = bsd
printcap name = /dev/null
disable spoolss = Yes

vfs objects = acl_xattr
map acl inherit = Yes
store dos attributes = Yes

username map = /usr/local/etc/samba/usermap.cnf

[ftpSD1]
        path = /storage/ftproot
        read only = no
        admin users = @"REDACTED.COM\Domain Admins"
        vfs objects = zfsacl,full_audit
        full_audit:prefix = %u|%I|%m|%S
        full_audit:success = opendir mkdir rmdir closedir open close read pread write pwrite rename unlink chmod fchmod chown fchown
        full_audit:failure = connect
        full_audit:facility = local7
        full_audit:priority = NOTICE


[smbSD1]
        path = /storage/smbroot
        read only = no
        admin users = @"REDACTED.COM\Domain Admins"
        vfs objects = zfsacl,full_audit
        full_audit:prefix = %u|%I|%m|%S
        full_audit:success = opendir mkdir rmdir closedir open close read pread write pwrite rename unlink chmod fchmod chown fchown fsetxattr fset_nt_acl
        full_audit:failure = connect
        full_audit:facility = local7
        full_audit:priority = NOTICE

        #:> cat /etc/rc.conf
        hostname="fs1.REDACTED.com"

        ifconfig_vtnet0="inet XX.XX.XX.XX/24"
        defaultrouter="XX.XX.XX.1"

        sshd_enable="YES"
        ntpd_enable="YES"

        sendmail_enable="NO"
        syslogd_enable="YES"
        syslogd_flags="-ss"

        samba_server_enable="YES"
        samba_server_config="/usr/local/etc/samba/smb4.conf"
        winbindd_enable="YES"

        squid_enable="YES"

        dumpdev="AUTO"
        zfs_enable="YES"

        qemu_guest_agent_enable="YES"

        rpcbind_enable="YES"
        nfs_server_enable="YES"
        mountd_flags="-r"
 
Hi SirDice,

thanks for the quick response! Do I need to rebuild the whole Samba ecosystem or only the pam_winbind? Is there any guide on how to do that here in the official forums?

The system booted fine after the update and shows the right version, am I missing somethig?

Bash:
[root@fs1 ~]# uname -a
FreeBSD fs1.REDACTED.com 12.2-RELEASE-p14 FreeBSD 12.2-RELEASE-p14 GENERIC  amd64


AFAIK no packages were build from source, does that mean that I am using the wrong repositories or the upgrade process had some "hicckups"? :)

Cheers!
 
I followed all the instructions here and rebooted the machine at the specified steps - https://www.freebsd.org/releases/12.2R/installation/

freebsd-update cannot find anything more to update, see below:
Bash:
[root@fs1 ~]# freebsd-update install
src component not installed, skipped
No updates are available to install.
Run '/usr/sbin/freebsd-update fetch' first.
[root@fs1 ~]# freebsd-update fetch
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p15.

WARNING: FreeBSD 12.2-RELEASE-p14 HAS PASSED ITS END-OF-LIFE DATE.
Any security issues discovered after Thu Mar 31 03:00:00 EEST 2022
will not have been corrected.

[root@fs1 ~]# freebsd-update install
src component not installed, skipped
No updates are available to install.
Run '/usr/sbin/freebsd-update fetch' first.

What am I missing? Do I need to upgrade to 12.3-RELEASE to get the updates? Is there any package I need to install to get the new pam_winbind.so ? Reinstall samba maybe?

Cheers!
 
Bash:
[root@fs1 ~]#  freebsd-version -ruk
12.2-RELEASE-p14
12.2-RELEASE-p14
12.2-RELEASE-p15

I will try with pkg upgrade now, any chance I can break something further :)

EDIT: Well, pkg says all is fine :)

Bash:
[root@fs1 /etc/pam.d]# pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.


[root@fs1 /etc/pam.d]# pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (19 candidates): 100%
Processing candidates (19 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
 
Also, is there a way to find out which package or series of packages the pam_winbind.so module / library belongs to, and reinstall only that package?
 
OK, so the only hope of fixing the PAM module is upgrading to 12.3?
Are the instructions here the right ones - https://www.freebsd.org/releases/12.3R/installation/
Yes, the procedure is pretty much the same for any minor version upgrade.

Also, is there a way to find out which package or series of packages the pam_winbind.so module / library belongs to, and reinstall only that package?
Code:
root@molly:~ # pkg which /usr/local/lib/pam_winbind.so
/usr/local/lib/pam_winbind.so was installed by package samba412-4.12.15_4

You can reinstall this all day long but it's not going to solve the issue. The issue isn't with pam_winbind.so. The issue is caused by an outdated base OS.
 
Packages are built for 12.2 and you're still on 12.1.
Apparently had a bit of a brainfart this morning, I'm going to blame it on a lack of caffeine and proper sleep. 12.3-RELEASE has already happened, 12.2-RELEASE is EoL. Packages are built for 12.3 now.
 
OK, so upgrading to 12.3 with the proper procedure will most likely solve the issue with pam_winbind.so, because it will also update the base OS, am I understanding this correctly :)

I'll start the upgrade to 12.3 in the meantime.

Cheers!
 
OK, so upgrading to 12.3 with the proper procedure will most likely solve the issue with pam_winbind.so, because it will also update the base OS, am I understanding this correctly
Not also, freebsd-update(8) will update/upgrade the base OS. Nothing else. Because you're new to FreeBSD you're not yet aware that we have a fairly strict separation of the "base OS" and additional, third-party, applications (ports/packages).
 
Good morning,

thanks for the clarifications SirDice, after the upgrade to 12.3 everything works fine, all the users can connect again to their shares and the permissions are as expected.

Many thanks to all, problem solved! :)

Wish you all a nice weekend!
 
Back
Top