Hi!
I can add another data point, possibly related, from a FreeBSD 14.4 system running on a Vultr KVM VPS.
I am not claiming that this proves a FreeBSD bug, a Vultr issue, or the same root cause reported by other users in this thread. I am sharing it because I have a locally reproducible, intermittent failure that may help narrow the investigation.
The initial visible symptom was this `sshd` fatal error:
Code:
fatal: pack_hostkeys: serialize hostkey private: string is too large
On my affected system, this occurred even after reducing the active SSH host key configuration to ED25519 only.
While investigating that, I found that the issue does not appear to be limited to `sshd`. I can also reproduce intermittent SIGSEGVs in base system utilities that perform passwd/group/NSS lookups, especially `getent`.
The strongest reproducer so far is a loop that repeatedly runs:
Code:
getent passwd root
getent group wheel
id root
The failures are intermittent, not deterministic. The affected host is a Vultr KVM VPS:
Code:
FreeBSD: 14.4-RELEASE-p7 amd64
Kernel/userland: 14.4-RELEASE-p7 / 14.4-RELEASE-p7
Virtualization: kvm
vCPU: 1
RAM: 1 GiB
Relevant loader/module information:
Code:
aesni_load="YES"
cc_htcp_load="YES"
crypto_load="YES"
# cryptodev_load="YES" # disabled temporarily while investigating recurring sshd crashes
virtio_random_load="YES"
Relevant loaded modules include:
Code:
virtio_random.ko
cc_htcp.ko
pf.ko
The system reports random sources similar to:
Code:
'VirtIO Entropy Adapter','Intel Secure Key RNG'
`cryptodev.ko` is not currently loaded on the affected host.
The effective `sshd` configuration was validated as root. Relevant lines:
Code:
permitrootlogin no
passwordauthentication no
kbdinteractiveauthentication no
hostkeyalgorithms ssh-ed25519
hostkey /etc/ssh/ssh_host_ed25519_key
subsystem sftp /usr/libexec/sftp-server
maxstartups 3:30:10
Only the ED25519 SSH host key is present under `/etc/ssh`. On the affected host, I simplified `/etc/nsswitch.conf` to avoid compat/NIS:
Code:
group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
protocols: files
rpc: files
Before testing, I also checked the local passwd/group databases:
Code:
pwd_mkdb -C /etc/master.passwd
chkgrp /etc/group
Both completed without errors.
The crash still reproduced after simplifying `nsswitch.conf` to `files`.
I have not yet tested the affected host with the default `/etc/nsswitch.conf`, which uses `compat` for some databases. If it would help distinguish whether the simplified `files`-only configuration is a trigger rather than a workaround, I can run the same reproducer with the default configuration as a control.
The reproducer I used is:
Code:
#!/bin/sh
ulimit -c unlimited
iteration=1
max_iterations=1000
result=0
while [ "$iteration" -le "$max_iterations" ]; do
echo "ITERATION $iteration: getent passwd root"
getent passwd root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent passwd root failed at iteration $iteration with rc=$rc"
result=10
break
fi
echo "ITERATION $iteration: getent group wheel"
getent group wheel >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent group wheel failed at iteration $iteration with rc=$rc"
result=12
break
fi
echo "ITERATION $iteration: id root"
id root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: id root failed at iteration $iteration with rc=$rc"
result=13
break
fi
iteration=$((iteration + 1))
done
if [ "$result" -eq 0 ]; then
echo "RESULT: completed $max_iterations iterations without failure"
else
echo "RESULT: reproducer failed with result=$result"
fi
exit "$result"
I also tested a separate comparison host.
Both systems were updated to the same FreeBSD release and patch level:
Code:
14.4-RELEASE-p7
14.4-RELEASE-p7
The comparison host is not identical to the affected host. It is a separate Hyper-V guest with more memory and different loaded modules:
Code:
Virtualization: hv
vCPU: 1
RAM: 4 GiB
Relevant loaded modules: cryptodev.ko, zfs.ko, mac_ntpd.ko
Random source: Intel Secure Key RNG
The comparison host also has a different `/etc/nsswitch.conf`; it still uses the default compat/NIS-style entries for some databases, while the affected host was simplified to `files`.
Therefore, this comparison only controls the FreeBSD release and patch level. It does not prove that all runtime conditions are identical.
With that limitation, I ran the same 1000-iteration reproducer five times on each host.
Results:
Code:
comparison host: 5/5 runs completed 1000 iterations without failure
affected host: 4/5 runs completed 1000 iterations without failure; 1/5 failed
The failed run on the affected host was:
Code:
FAILURE: getent group wheel failed at iteration 679 with rc=139
Segmentation fault (core dumped)
RESULT: reproducer failed with result=12
The backtrace from that `getent` core was:
Code:
* thread #1, name = 'getent', stop reason = signal SIGSEGV
* frame #0: libc.so.7`strcasecmp_l + 118
frame #1: libc.so.7`bsearch + 70
frame #2: libc.so.7`_nsdbtaddsrc + 139
frame #3: libc.so.7`_nsyyparse + 1099
frame #4: libc.so.7`_nsdispatch + 976
frame #5: libc.so.7`setgroupent + 44
frame #6: getent
The presence of `bsearch` in the backtrace, combined with `_nsdbtaddsrc`, suggests that the crash may occur while the NSS subsystem is traversing or modifying its internal source list. This could point to heap corruption affecting that structure, rather than a simple parsing error.
The loaded images in that core were only:
Code:
/usr/bin/getent
/lib/libc.so.7
/libexec/ld-elf.so.1
Earlier controlled runs on the same affected host showed similar behavior. One run failed in `getent group wheel` at iteration 581:
Code:
FAILURE: getent group wheel failed at iteration 581 with rc=139
Segmentation fault (core dumped)
Another run failed at iteration 172:
Code:
ITERATION 172: getent passwd root
ITERATION 172: getent group wheel
Segmentation fault (core dumped)
FAILURE: getent group wheel failed at iteration 172 with rc=139
RESULT: reproducer failed with result=12
Earlier observations on the same affected system also included crashes in `getent passwd root`, `id root`, `env`, and `newsyslog`. The backtraces consistently converge on libc/NSS-related paths such as:
Code:
libc.so.7`strcasecmp_l
libc.so.7`bsearch
libc.so.7`_nsdbtaddsrc
libc.so.7`_nsyyparse
libc.so.7`_nsdispatch
setgroupent / setpassent / getpwnam
In my case, the `sshd` failure may be only one visible symptom of broader userland instability. The local `getent`/`id` reproducer does not depend on external SSH traffic.
The issue is intermittent. The comparison host running the same FreeBSD release and patch level completed all fresh runs successfully, while the affected Vultr KVM host failed in one of five fresh runs.
This suggests that the issue is not a deterministic failure of FreeBSD 14.4-RELEASE-p7 alone. The differences that now look most relevant to compare are:
Code:
KVM/Vultr versus Hyper-V
VirtIO Entropy Adapter
1 GiB RAM versus 4 GiB RAM
loaded kernel modules
provider image versus clean FreeBSD ISO installation
local system state
One visible difference is that the affected Vultr KVM host uses the VirtIO Entropy Adapter, while the comparison Hyper-V host does not. I have not disabled `virtio_random` on the affected production host yet, but I can test that on a clone or clean test instance if it would help.
I have not tested the reproducer in single-user mode yet. I can do that if it would help isolate userland/NSS behavior from normal multi-user services, SSH traffic, cron, and other background activity.
The current backtraces do not include source line numbers because the matching libc debug symbols are not installed. If line-number backtraces would be useful, I can install the matching debug symbols and rerun the reproducer.
Based on the current evidence, the most logical next steps seem to be:
Code:
1. Deploy a fresh Vultr instance from the official FreeBSD ISO and run the reproducer.
2. If the failure still occurs, test with virtio_random disabled.
3. If it persists, attempt reproduction on a non-Vultr KVM or bhyve VM.
4. Optionally test the affected host with the default nsswitch.conf as another control.
5. Optionally test in single-user mode to reduce background activity.
So, I would appreciate suggestions on what to test next.
I am willing to run additional tests, provide sanitized logs, rerun the reproducer with suggested changes, or collect more specific diagnostics if that would help. I am intentionally not posting raw core files publicly, but I can keep them available privately if a FreeBSD developer needs them through an appropriate channel.
Thanks in advance for any guidance.
Trapo
