ldap issues on FreeBSD 14.0

dvl@

dvl@

Developer
We've updated two servers to FreeBSD 14 (from 13.2) and neither one is coming to the ldap party.

ldapsearch works after a kinit.

We know nslcd started on the command line will segfault after someone tries to log in.

We've got lots of core files for kstart and nslcd. Both are segfautling at the same memory address.

We looked in /etc/pam.d/ just because: nothing different in there.

Both hosts are using net/nss-pam-ldapd-sasl-0.9.12_2 (which worked OK before the upgrade to 14.0).

Ideas please
 
More news.

Scroll all the way to the right on frame #0 and you'll see at evp_lib.c:406:27

Code: 
root@walnuts-dev:/tmp # lldb --core 0.k5start.98404.core /usr/local/bin/k5start
(lldb) target create "/usr/local/bin/k5start" --core "0.k5start.98404.core"
Core file '/tmp/0.k5start.98404.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'k5start', stop reason = signal SIGSEGV
  * frame #0: 0x0000000825446c7f libcrypto.so.30`EVP_Cipher(ctx=0x00000a6d97eb4d80, out="\xba\U00000004\x82Q\x96\x99{\xd00\U0000001a\xa0\U00000011\U00000018\U0000000f20231214010802Z\xa1\U00000005\U00000002\U00000003\f[X\xb6:ǖ\xff\xff\xff\U0000001f", in="\xba\U00000004\x82Q\x96\x99{\xd00\U0000001a\xa0\U00000011\U00000018\U0000000f20231214010802Z\xa1\U00000005\U00000002\U00000003\f[X\xb6:ǖ\xff\xff\xff\U0000001f", inl=36) at evp_lib.c:406:27
    frame #1: 0x00000008236be4b6 libkrb5.so.11`ARCFOUR_encrypt [inlined] ARCFOUR_subencrypt(context=0x00000a6d97e1b000, key=<unavailable>, data=0x00000a6d97ed14c0, len=<unavailable>, usage=<unavailable>, ivec=<unavailable>) at crypto-arcfour.c:184:5
    frame #2: 0x00000008236be352 libkrb5.so.11`ARCFOUR_encrypt(context=0x00000a6d97e1b000, key=<unavailable>, data=0x00000a6d97ed14c0, len=<unavailable>, encryptp=<unavailable>, usage=<unavailable>, ivec=<unavailable>) at crypto-arcfour.c:311:9
    frame #3: 0x00000008236c2ac8 libkrb5.so.11`krb5_encrypt_ivec [inlined] encrypt_internal_special(context=0x00000a6d97e1b000, crypto=0x00000a6d97e2d8f0, usage=1, data=0x00000a6d97ecf920, len=28, result=0x0000000820f32d08, ivec=<unavailable>) at crypto.c:969:11
    frame #4: 0x00000008236c2a39 libkrb5.so.11`krb5_encrypt_ivec(context=0x00000a6d97e1b000, crypto=0x00000a6d97e2d8f0, usage=1, data=0x00000a6d97ecf920, len=28, result=0x0000000820f32d08, ivec=0x0000000000000000) at crypto.c:1761:9
    frame #5: 0x00000008236c2e65 libkrb5.so.11`krb5_encrypt_EncryptedData [inlined] krb5_encrypt(context=<unavailable>, crypto=0x00000a6d97e2d8f0, usage=<unavailable>, data=<unavailable>, len=<unavailable>, result=0x0000000820f32d08) at crypto.c:1775:12
    frame #6: 0x00000008236c2e52 libkrb5.so.11`krb5_encrypt_EncryptedData(context=0x00000a6d97e1b000, crypto=0x00000a6d97e2d8f0, usage=1, data=0x00000a6d97ecf920, len=<unavailable>, kvno=0, result=0x0000000820f32cf8) at crypto.c:1793:12
    frame #7: 0x00000008236d32c5 libkrb5.so.11`add_enc_ts_padata [inlined] make_pa_enc_timestamp(context=0x00000a6d97e1b000, md=0x00000a6d97ed5000, etype=<unavailable>, key=0x00000a6d97ecfa00) at init_creds_pw.c:954:11
    frame #8: 0x00000008236d3211 libkrb5.so.11`add_enc_ts_padata(context=0x00000a6d97e1b000, md=0x00000a6d97ed5000, client=<unavailable>, keyproc=(libkrb5.so.11`keytab_key_proc at init_creds_pw.c:1479), keyseed=0x00000a6d97e20180, enctypes=0x00000a6d97e2d8c0, netypes=1, salt=0x00000a6d97e2d8c8, s2kparams=0x0000000000000000) at init_creds_pw.c:1018:8
    frame #9: 0x00000008236d15da libkrb5.so.11`krb5_init_creds_step [inlined] pa_data_to_md_ts_enc(context=0x00000a6d97e1b000, a=0x00000a6d97e5a0d0, client=<unavailable>, ctx=0x00000a6d97e5a000, ppaid=0x00000a6d97e2d8c0, md=0x00000a6d97ed5000) at init_creds_pw.c:1040:2
    frame #10: 0x00000008236d15a6 libkrb5.so.11`krb5_init_creds_step at init_creds_pw.c:1224:2
    frame #11: 0x00000008236d156d libkrb5.so.11`krb5_init_creds_step(context=0x00000a6d97e1b000, ctx=0x00000a6d97e5a000, in=0x0000000820f33028, out=0x0000000820f33018, hostinfo=0x0000000000000000, flags=0x0000000820f33014) at init_creds_pw.c:1816:11
    frame #12: 0x00000008236d1caf libkrb5.so.11`krb5_init_creds_get(context=0x00000a6d97e1b000, ctx=0x00000a6d97e5a000) at init_creds_pw.c:1928:8
    frame #13: 0x00000008236d29de libkrb5.so.11`krb5_get_init_creds_keytab(context=0x00000a6d97e1b000, creds=0x0000000820f33100, client=<unavailable>, keytab=0x00000a6d97e2f000, start_time=<unavailable>, in_tkt_service="krbtgt/ABC.EXAMPLE.COM@ABC.EXAMPLE.COM", options=0x00000a6d97e10440) at init_creds_pw.c:2136:11
    frame #14: 0x00000000002068f2 k5start`___lldb_unnamed_symbol208 + 690
    frame #15: 0x00000000002053e6 k5start`___lldb_unnamed_symbol202 + 182
    frame #16: 0x0000000000206609 k5start`___lldb_unnamed_symbol207 + 2617
    frame #17: 0x00000008287f7afa libc.so.7`__libc_start1(argc=11, argv=0x0000000820f338c0, env=0x0000000820f33920, cleanup=<unavailable>, mainX=(k5start`___lldb_unnamed_symbol207)) at libc_start1.c:157:7
    frame #18: 0x0000000000205240 k5start`___lldb_unnamed_symbol198 + 48
 
Last edited:
Above, we thought kstart was broken. It goes deeper I think. Here, msktutil also dump:

This is on another host:

Code: 
bsdpackage01p# lldb --core msktutil.core /usr/local/sbin/msktutil
(lldb) target create "/usr/local/sbin/msktutil" --core "msktutil.core"
Core file '/tmp/msktutil.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'msktutil', stop reason = signal SIGSEGV
  * frame #0: 0x0000000000000000
    frame #1: 0x00000008235f7308 libkrb5.so.11`___lldb_unnamed_symbol2440 + 312
    frame #2: 0x00000008235f83f5 libkrb5.so.11`krb5_string_to_key_data_salt + 277
    frame #3: 0x00000000002114bc msktutil`___lldb_unnamed_symbol450 + 140
    frame #4: 0x000000000021a4ad msktutil`___lldb_unnamed_symbol506 + 3293
    frame #5: 0x0000000000219601 msktutil`___lldb_unnamed_symbol505 + 129
    frame #6: 0x0000000000214644 msktutil`___lldb_unnamed_symbol478 + 1172
    frame #7: 0x0000000000215bb4 msktutil`___lldb_unnamed_symbol481 + 4004
    frame #8: 0x000000082a8e8afa libc.so.7`__libc_start1 + 298
    frame #9: 0x0000000000210bb0 msktutil`___lldb_unnamed_symbol436 + 48
 
Code: 
(gdb) core /tmp/msktutil.core
[New LWP 106960]
Core was generated by `msktutil --create -b OU=xxx,OU=xxx,OU=xxx,OU=xxx --computer-name'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x0000000000000000 in ?? ()


Code: 
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000000082376a308 in ?? ()
#2  0x5b00000800000017 in ?? ()
#3  0x000000000000003f in ?? ()
#4  0xdc81a55c6cd0fa40 in ?? ()
#5  0x00001cf971c1b000 in ?? ()
#6  0x0000000000000000 in ?? ()
 
Here is more information on each frame:
Code: 
(lldb) frame select 1
frame #1: 0x000000082376a308 libkrb5.so.11`___lldb_unnamed_symbol2440 + 312
libkrb5.so.11`___lldb_unnamed_symbol2440:
->  0x82376a308 <+312>: movzbl 0x1(%r15,%r13,2), %eax
    0x82376a30e <+318>: movb   %al, -0x39(%rbp)
    0x82376a311 <+321>: movl   $0x1, %edx
    0x82376a316 <+326>: movq   %rbx, %rdi
(lldb) frame select 2
frame #2: 0x000000082376b3f5 libkrb5.so.11`krb5_string_to_key_data_salt + 277
libkrb5.so.11`krb5_string_to_key_data_salt:
->  0x82376b3f5 <+277>: movl   %eax, %r15d
    0x82376b3f8 <+280>: jmp    0x6f3a5                   ; <+197>
    0x82376b3fa <+282>: callq  0x77190                   ; symbol stub for: __stack_chk_fail
    0x82376b3ff:        nop
(lldb) frame select 3
frame #3: 0x00000000002114bc msktutil`___lldb_unnamed_symbol450 + 140
msktutil`___lldb_unnamed_symbol450:
->  0x2114bc <+140>: testl  %eax, %eax
    0x2114be <+142>: jne    0x2114eb                  ; <+187>
    0x2114c0 <+144>: leaq   -0x40(%rbp), %rcx
    0x2114c4 <+148>: movq   %r15, %rdi
(lldb) frame select 4
frame #4: 0x000000000021a4ad msktutil`___lldb_unnamed_symbol506 + 3293
msktutil`___lldb_unnamed_symbol506:
->  0x21a4ad <+3293>: incq   %r13
    0x21a4b0 <+3296>: movq   -0x98(%rbp), %rax
    0x21a4b7 <+3303>: subq   %rbx, %rax
    0x21a4ba <+3306>: sarq   $0x2, %rax
(lldb) frame select 5
frame #5: 0x0000000000219601 msktutil`___lldb_unnamed_symbol505 + 129
msktutil`___lldb_unnamed_symbol505:
->  0x219601 <+129>: cmpb   $0x0, 0x259(%rbx)
    0x219608 <+136>: jne    0x219619                  ; <+153>
    0x21960a <+138>: leaq   0x138(%rbx), %rdi
    0x219611 <+145>: movq   %rbx, %rsi
(lldb) frame select 6
frame #6: 0x0000000000214644 msktutil`___lldb_unnamed_symbol478 + 1172
msktutil`___lldb_unnamed_symbol478:
->  0x214644 <+1172>: movq   %rbx, %rdi
    0x214647 <+1175>: callq  0x218940                  ; ___lldb_unnamed_symbol504
    0x21464c <+1180>: movq   %rbx, %rdi
    0x21464f <+1183>: callq  0x214aa0                  ; ___lldb_unnamed_symbol479
(lldb) frame select 7
frame #7: 0x0000000000215bb4 msktutil`___lldb_unnamed_symbol481 + 4004
msktutil`___lldb_unnamed_symbol481:
->  0x215bb4 <+4004>: movl   %eax, %ebx
    0x215bb6 <+4006>: jmp    0x215bf0                  ; <+4064>
    0x215bb8 <+4008>: movq   0x191d9(%rip), %rcx       ; __stderrp
    0x215bbf <+4015>: movl   $0x209d66, %edi           ; imm = 0x209D66
(lldb) frame select 8
frame #8: 0x000000082bf95afa libc.so.7`__libc_start1 + 298
libc.so.7`__libc_start1:
->  0x82bf95afa <+298>: movl   %eax, %edi
    0x82bf95afc <+300>: callq  0x1c9200                  ; symbol stub for: exit
    0x82bf95b01:        nopw   %cs:(%rax,%rax)
libc.so.7`__libc_start1_gcrt:
    0x82bf95b10 <+0>:   pushq  %rbp
(lldb) frame select 9
frame #9: 0x0000000000210bb0 msktutil`___lldb_unnamed_symbol436 + 48
msktutil`___lldb_unnamed_symbol436:
->  0x210bb0 <+48>: int3
    0x210bb1 <+49>: int3
    0x210bb2 <+50>: int3
    0x210bb3 <+51>: int3
 
For now, we're going back to FreeBSD 13.3 (due to be released soon) and abandoning 14.0 for use with Active Directory for now. It seems like some utilities need to be patched up to work with the newer OpenSSL.
 
dvl@ , update to 14-STABLE. The fix to this has been MFCed January 22. The patch was not MFSed to 14.0-RELEASE but will be in 14.1-RELEASE. You can read the gory details in PR/272835.
 
cy@ said:
dvl@ , update to 14-STABLE. The fix to this has been MFCed January 22. The patch was not MFSed to 14.0-RELEASE but will be in 14.1-RELEASE. You can read the gory details in PR/272835.
Click to expand...
-STABLE is not an option for us. We use -RELEASE.
 
I tried building with the indicated options.

Code: 
[dvl@ava-pkg-02prd:~] $ pkg info msktutil | grep GSS
        GSSAPI_BASE    : off
        GSSAPI_HEIMDAL : off
        GSSAPI_MIT     : on

FWIW,
msktutil dies sooner (right after SASL/GSS-SPNEGO authentication started) as opposed to much later (during
-- add_principal_keytab: Adding entry of enctype 0x17):

Code: 
$ kinit dvl@[redacted]
dvl@[redacted]'s Password:
$ /usr/local/sbin/msktutil -c -b OU=FOOBAR-Hosts --server dc01.[redacted] --user-creds-only --verbose
 -- init_password: Wiping the computer password structure
 -- generate_new_password: Generating a new, random password for the computer account
 -- generate_new_password: Characters read from /dev/urandom: 79
 -- get_default_keytab: Obtaining the default keytab name: FILE:/etc/krb5.keytab
 -- create_fake_krb5_conf: Created fake krb5.conf file: /tmp/.msktkrb5.conf-185KO2
 -- destroy_g_context: Destroying Kerberos context
 -- initialize_g_context: Creating Kerberos context
 -- get_default_samaccountname: Determined sAMAccountName: ava-pkg-02prd
 -- finalize_exec: sAMAccountName: ava-pkg-02prd$
 -- get_short_hostname: Determined short hostname: ava-pkg-02prd
 -- get_short_hostname: Determined short hostname: ava-pkg-02prd
 -- try_user_creds: Checking if default ticket cache has tickets
 -- finalize_exec: Authenticated using method 5
 -- LDAPConnection: Connecting to LDAP server: dc01.[redacted]
SASL/GSS-SPNEGO authentication started
Segmentation fault (core dumped)
[dvl@ava-pkg-02prd:/usr/local/etc/poudriere.d] $
 
Historically, I do not know. We're run -RELEASE for more than 10 years. We patch via freebsd-update fetch install. If I had to guess, it'd be because patching the OS is rather straight forward.

We have about 650 servers and they are all different. There might be a cluster where they all do the similar tasks, but they are mostly doing different things.
 
FWIW, at home, I once followed -STABLE. However, once freebsd-update became a thing, I've followed -RELEASE.
 
After patching our 14.0 hosts to p6, I can run kstart again. nslcd is working too.

Other ports are also working since we are building with MIT version of heimdal. This can be closed.
 
You must log in or register to reply here.
Back
Top