sshd frequent crashes (segfault and fatal during key checks??)

Hi!

I can add another data point, possibly related, from a FreeBSD 14.4 system running on a Vultr KVM VPS.

I am not claiming that this proves a FreeBSD bug, a Vultr issue, or the same root cause reported by other users in this thread. I am sharing it because I have a locally reproducible, intermittent failure that may help narrow the investigation.

The initial visible symptom was this `sshd` fatal error:

Code: 
fatal: pack_hostkeys: serialize hostkey private: string is too large

On my affected system, this occurred even after reducing the active SSH host key configuration to ED25519 only.

While investigating that, I found that the issue does not appear to be limited to `sshd`. I can also reproduce intermittent SIGSEGVs in base system utilities that perform passwd/group/NSS lookups, especially `getent`.

The strongest reproducer so far is a loop that repeatedly runs:

Code: 
getent passwd root
getent group wheel
id root

The failures are intermittent, not deterministic. The affected host is a Vultr KVM VPS:

Code: 
FreeBSD: 14.4-RELEASE-p7 amd64
Kernel/userland: 14.4-RELEASE-p7 / 14.4-RELEASE-p7
Virtualization: kvm
vCPU: 1
RAM: 1 GiB

Relevant loader/module information:

Code: 
aesni_load="YES"
cc_htcp_load="YES"
crypto_load="YES"

# cryptodev_load="YES"  # disabled temporarily while investigating recurring sshd crashes

virtio_random_load="YES"

Relevant loaded modules include:

Code: 
virtio_random.ko
cc_htcp.ko
pf.ko

The system reports random sources similar to:

Code: 
'VirtIO Entropy Adapter','Intel Secure Key RNG'

`cryptodev.ko` is not currently loaded on the affected host.

The effective `sshd` configuration was validated as root. Relevant lines:

Code: 
permitrootlogin no
passwordauthentication no
kbdinteractiveauthentication no
hostkeyalgorithms ssh-ed25519
hostkey /etc/ssh/ssh_host_ed25519_key
subsystem sftp /usr/libexec/sftp-server
maxstartups 3:30:10

Only the ED25519 SSH host key is present under `/etc/ssh`. On the affected host, I simplified `/etc/nsswitch.conf` to avoid compat/NIS:

Code: 
group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
protocols: files
rpc: files

Before testing, I also checked the local passwd/group databases:

Code: 
pwd_mkdb -C /etc/master.passwd
chkgrp /etc/group

Both completed without errors.

The crash still reproduced after simplifying `nsswitch.conf` to `files`.

I have not yet tested the affected host with the default `/etc/nsswitch.conf`, which uses `compat` for some databases. If it would help distinguish whether the simplified `files`-only configuration is a trigger rather than a workaround, I can run the same reproducer with the default configuration as a control.

The reproducer I used is:

Code: 
#!/bin/sh

ulimit -c unlimited

iteration=1
max_iterations=1000
result=0

while [ "$iteration" -le "$max_iterations" ]; do
echo "ITERATION $iteration: getent passwd root"
getent passwd root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent passwd root failed at iteration $iteration with rc=$rc"
result=10
break
fi

echo "ITERATION $iteration: getent group wheel"
getent group wheel >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent group wheel failed at iteration $iteration with rc=$rc"
result=12
break
fi

echo "ITERATION $iteration: id root"
id root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: id root failed at iteration $iteration with rc=$rc"
result=13
break
fi

iteration=$((iteration + 1))
done

if [ "$result" -eq 0 ]; then
echo "RESULT: completed $max_iterations iterations without failure"
else
echo "RESULT: reproducer failed with result=$result"
fi

exit "$result"

I also tested a separate comparison host.

Both systems were updated to the same FreeBSD release and patch level:

Code: 
14.4-RELEASE-p7
14.4-RELEASE-p7

The comparison host is not identical to the affected host. It is a separate Hyper-V guest with more memory and different loaded modules:

Code: 
Virtualization: hv
vCPU: 1
RAM: 4 GiB
Relevant loaded modules: cryptodev.ko, zfs.ko, mac_ntpd.ko
Random source: Intel Secure Key RNG

The comparison host also has a different `/etc/nsswitch.conf`; it still uses the default compat/NIS-style entries for some databases, while the affected host was simplified to `files`.

Therefore, this comparison only controls the FreeBSD release and patch level. It does not prove that all runtime conditions are identical.

With that limitation, I ran the same 1000-iteration reproducer five times on each host.

Results:

Code: 
comparison host: 5/5 runs completed 1000 iterations without failure
affected host:   4/5 runs completed 1000 iterations without failure; 1/5 failed

The failed run on the affected host was:

Code: 
FAILURE: getent group wheel failed at iteration 679 with rc=139
Segmentation fault (core dumped)
RESULT: reproducer failed with result=12

The backtrace from that `getent` core was:

Code: 
* thread #1, name = 'getent', stop reason = signal SIGSEGV

  * frame #0: libc.so.7`strcasecmp_l + 118
    frame #1: libc.so.7`bsearch + 70
    frame #2: libc.so.7`_nsdbtaddsrc + 139
    frame #3: libc.so.7`_nsyyparse + 1099
    frame #4: libc.so.7`_nsdispatch + 976
    frame #5: libc.so.7`setgroupent + 44
    frame #6: getent

The presence of `bsearch` in the backtrace, combined with `_nsdbtaddsrc`, suggests that the crash may occur while the NSS subsystem is traversing or modifying its internal source list. This could point to heap corruption affecting that structure, rather than a simple parsing error.

The loaded images in that core were only:

Code: 
/usr/bin/getent
/lib/libc.so.7
/libexec/ld-elf.so.1

Earlier controlled runs on the same affected host showed similar behavior. One run failed in `getent group wheel` at iteration 581:

Code: 
FAILURE: getent group wheel failed at iteration 581 with rc=139
Segmentation fault (core dumped)

Another run failed at iteration 172:

Code: 
ITERATION 172: getent passwd root
ITERATION 172: getent group wheel
Segmentation fault (core dumped)
FAILURE: getent group wheel failed at iteration 172 with rc=139
RESULT: reproducer failed with result=12

Earlier observations on the same affected system also included crashes in `getent passwd root`, `id root`, `env`, and `newsyslog`. The backtraces consistently converge on libc/NSS-related paths such as:

Code: 
libc.so.7`strcasecmp_l
libc.so.7`bsearch
libc.so.7`_nsdbtaddsrc
libc.so.7`_nsyyparse
libc.so.7`_nsdispatch
setgroupent / setpassent / getpwnam

In my case, the `sshd` failure may be only one visible symptom of broader userland instability. The local `getent`/`id` reproducer does not depend on external SSH traffic.

The issue is intermittent. The comparison host running the same FreeBSD release and patch level completed all fresh runs successfully, while the affected Vultr KVM host failed in one of five fresh runs.

This suggests that the issue is not a deterministic failure of FreeBSD 14.4-RELEASE-p7 alone. The differences that now look most relevant to compare are:

Code: 
KVM/Vultr versus Hyper-V
VirtIO Entropy Adapter
1 GiB RAM versus 4 GiB RAM
loaded kernel modules
provider image versus clean FreeBSD ISO installation
local system state

One visible difference is that the affected Vultr KVM host uses the VirtIO Entropy Adapter, while the comparison Hyper-V host does not. I have not disabled `virtio_random` on the affected production host yet, but I can test that on a clone or clean test instance if it would help.

I have not tested the reproducer in single-user mode yet. I can do that if it would help isolate userland/NSS behavior from normal multi-user services, SSH traffic, cron, and other background activity.

The current backtraces do not include source line numbers because the matching libc debug symbols are not installed. If line-number backtraces would be useful, I can install the matching debug symbols and rerun the reproducer.

Based on the current evidence, the most logical next steps seem to be:

Code: 
1. Deploy a fresh Vultr instance from the official FreeBSD ISO and run the reproducer.
2. If the failure still occurs, test with virtio_random disabled.
3. If it persists, attempt reproduction on a non-Vultr KVM or bhyve VM.
4. Optionally test the affected host with the default nsswitch.conf as another control.
5. Optionally test in single-user mode to reduce background activity.


So, I would appreciate suggestions on what to test next.

I am willing to run additional tests, provide sanitized logs, rerun the reproducer with suggested changes, or collect more specific diagnostics if that would help. I am intentionally not posting raw core files publicly, but I can keep them available privately if a FreeBSD developer needs them through an appropriate channel.

Thanks in advance for any guidance.

Trapo
 
Hmmm...

So I recall seeing this type of non-deterministic behaviour in the console login program (Alt+Fn). At that time, I think I was on a 14.x on a BETA, and I assumed I was fatfingering my password (keyboard map switch). But on a hunch after turning on core dumps for setuid binaries (sysctl kern.sugid_coredump=1) , I did actually get a login.core file. Which ofcourse I do not have anylonger. This would imply some sort of instability in that code path.

So maybe getting the dbg sysmbols in base installed and having that sysctl turned on should be considered. And then I guess "wait" for core to drop.
 
siladi said:
So maybe getting the dbg sysmbols in base installed and having that sysctl turned on should be considered. And then I guess "wait" for core to drop.
Click to expand...

The debug symbols don't have to be present at coredump time. Only at debugger time. Can even be on a different machine if distribution sets match.
 
OK guys, got a login.core file, 13M. Backtrace below:

Code: 
# gdb -q login
Reading symbols from login...
Reading symbols from /usr/lib/debug//usr/bin/login.debug...
(gdb) core /root/login.core
[New LWP 100369]
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
Core was generated by `login'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x0000098f6e706780 in ?? ()
(gdb) bt
#0  0x0000098f6e706780 in ?? ()
#1  0x0000098f749caa0d in thr_fork_impl (a=0x98f66021a48) at /usr/src/lib/libthr/thread/thr_fork.c:179
#2  0x0000098f749ca988 in __thr_fork () at /usr/src/lib/libthr/thread/thr_fork.c:319
#3  0x0000098745275eaf in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/usr.bin/login/login.c:548
(gdb)

Would be useful if OP or anyone else can get some core files. trapo did you say you got a reproducer ? turn on core dumps for setuid binaries (sysctl kern.sugid_coredump=1) and see if you get something useful.
 
You must log in or register to reply here.
Back
Top