sshd frequent crashes (segfault and fatal during key checks??)

Hi!

I can add another data point, possibly related, from a FreeBSD 14.4 system running on a Vultr KVM VPS.

I am not claiming that this proves a FreeBSD bug, a Vultr issue, or the same root cause reported by other users in this thread. I am sharing it because I have a locally reproducible, intermittent failure that may help narrow the investigation.

The initial visible symptom was this `sshd` fatal error:

Code: 
fatal: pack_hostkeys: serialize hostkey private: string is too large

On my affected system, this occurred even after reducing the active SSH host key configuration to ED25519 only.

While investigating that, I found that the issue does not appear to be limited to `sshd`. I can also reproduce intermittent SIGSEGVs in base system utilities that perform passwd/group/NSS lookups, especially `getent`.

The strongest reproducer so far is a loop that repeatedly runs:

Code: 
getent passwd root
getent group wheel
id root

The failures are intermittent, not deterministic. The affected host is a Vultr KVM VPS:

Code: 
FreeBSD: 14.4-RELEASE-p7 amd64
Kernel/userland: 14.4-RELEASE-p7 / 14.4-RELEASE-p7
Virtualization: kvm
vCPU: 1
RAM: 1 GiB

Relevant loader/module information:

Code: 
aesni_load="YES"
cc_htcp_load="YES"
crypto_load="YES"

# cryptodev_load="YES"  # disabled temporarily while investigating recurring sshd crashes

virtio_random_load="YES"

Relevant loaded modules include:

Code: 
virtio_random.ko
cc_htcp.ko
pf.ko

The system reports random sources similar to:

Code: 
'VirtIO Entropy Adapter','Intel Secure Key RNG'

`cryptodev.ko` is not currently loaded on the affected host.

The effective `sshd` configuration was validated as root. Relevant lines:

Code: 
permitrootlogin no
passwordauthentication no
kbdinteractiveauthentication no
hostkeyalgorithms ssh-ed25519
hostkey /etc/ssh/ssh_host_ed25519_key
subsystem sftp /usr/libexec/sftp-server
maxstartups 3:30:10

Only the ED25519 SSH host key is present under `/etc/ssh`. On the affected host, I simplified `/etc/nsswitch.conf` to avoid compat/NIS:

Code: 
group: files
hosts: files dns
networks: files
passwd: files
shells: files
services: files
protocols: files
rpc: files

Before testing, I also checked the local passwd/group databases:

Code: 
pwd_mkdb -C /etc/master.passwd
chkgrp /etc/group

Both completed without errors.

The crash still reproduced after simplifying `nsswitch.conf` to `files`.

I have not yet tested the affected host with the default `/etc/nsswitch.conf`, which uses `compat` for some databases. If it would help distinguish whether the simplified `files`-only configuration is a trigger rather than a workaround, I can run the same reproducer with the default configuration as a control.

The reproducer I used is:

Code: 
#!/bin/sh

ulimit -c unlimited

iteration=1
max_iterations=1000
result=0

while [ "$iteration" -le "$max_iterations" ]; do
echo "ITERATION $iteration: getent passwd root"
getent passwd root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent passwd root failed at iteration $iteration with rc=$rc"
result=10
break
fi

echo "ITERATION $iteration: getent group wheel"
getent group wheel >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: getent group wheel failed at iteration $iteration with rc=$rc"
result=12
break
fi

echo "ITERATION $iteration: id root"
id root >/dev/null
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAILURE: id root failed at iteration $iteration with rc=$rc"
result=13
break
fi

iteration=$((iteration + 1))
done

if [ "$result" -eq 0 ]; then
echo "RESULT: completed $max_iterations iterations without failure"
else
echo "RESULT: reproducer failed with result=$result"
fi

exit "$result"

I also tested a separate comparison host.

Both systems were updated to the same FreeBSD release and patch level:

Code: 
14.4-RELEASE-p7
14.4-RELEASE-p7

The comparison host is not identical to the affected host. It is a separate Hyper-V guest with more memory and different loaded modules:

Code: 
Virtualization: hv
vCPU: 1
RAM: 4 GiB
Relevant loaded modules: cryptodev.ko, zfs.ko, mac_ntpd.ko
Random source: Intel Secure Key RNG

The comparison host also has a different `/etc/nsswitch.conf`; it still uses the default compat/NIS-style entries for some databases, while the affected host was simplified to `files`.

Therefore, this comparison only controls the FreeBSD release and patch level. It does not prove that all runtime conditions are identical.

With that limitation, I ran the same 1000-iteration reproducer five times on each host.

Results:

Code: 
comparison host: 5/5 runs completed 1000 iterations without failure
affected host:   4/5 runs completed 1000 iterations without failure; 1/5 failed

The failed run on the affected host was:

Code: 
FAILURE: getent group wheel failed at iteration 679 with rc=139
Segmentation fault (core dumped)
RESULT: reproducer failed with result=12

The backtrace from that `getent` core was:

Code: 
* thread #1, name = 'getent', stop reason = signal SIGSEGV

  * frame #0: libc.so.7`strcasecmp_l + 118
    frame #1: libc.so.7`bsearch + 70
    frame #2: libc.so.7`_nsdbtaddsrc + 139
    frame #3: libc.so.7`_nsyyparse + 1099
    frame #4: libc.so.7`_nsdispatch + 976
    frame #5: libc.so.7`setgroupent + 44
    frame #6: getent

The presence of `bsearch` in the backtrace, combined with `_nsdbtaddsrc`, suggests that the crash may occur while the NSS subsystem is traversing or modifying its internal source list. This could point to heap corruption affecting that structure, rather than a simple parsing error.

The loaded images in that core were only:

Code: 
/usr/bin/getent
/lib/libc.so.7
/libexec/ld-elf.so.1

Earlier controlled runs on the same affected host showed similar behavior. One run failed in `getent group wheel` at iteration 581:

Code: 
FAILURE: getent group wheel failed at iteration 581 with rc=139
Segmentation fault (core dumped)

Another run failed at iteration 172:

Code: 
ITERATION 172: getent passwd root
ITERATION 172: getent group wheel
Segmentation fault (core dumped)
FAILURE: getent group wheel failed at iteration 172 with rc=139
RESULT: reproducer failed with result=12

Earlier observations on the same affected system also included crashes in `getent passwd root`, `id root`, `env`, and `newsyslog`. The backtraces consistently converge on libc/NSS-related paths such as:

Code: 
libc.so.7`strcasecmp_l
libc.so.7`bsearch
libc.so.7`_nsdbtaddsrc
libc.so.7`_nsyyparse
libc.so.7`_nsdispatch
setgroupent / setpassent / getpwnam

In my case, the `sshd` failure may be only one visible symptom of broader userland instability. The local `getent`/`id` reproducer does not depend on external SSH traffic.

The issue is intermittent. The comparison host running the same FreeBSD release and patch level completed all fresh runs successfully, while the affected Vultr KVM host failed in one of five fresh runs.

This suggests that the issue is not a deterministic failure of FreeBSD 14.4-RELEASE-p7 alone. The differences that now look most relevant to compare are:

Code: 
KVM/Vultr versus Hyper-V
VirtIO Entropy Adapter
1 GiB RAM versus 4 GiB RAM
loaded kernel modules
provider image versus clean FreeBSD ISO installation
local system state

One visible difference is that the affected Vultr KVM host uses the VirtIO Entropy Adapter, while the comparison Hyper-V host does not. I have not disabled `virtio_random` on the affected production host yet, but I can test that on a clone or clean test instance if it would help.

I have not tested the reproducer in single-user mode yet. I can do that if it would help isolate userland/NSS behavior from normal multi-user services, SSH traffic, cron, and other background activity.

The current backtraces do not include source line numbers because the matching libc debug symbols are not installed. If line-number backtraces would be useful, I can install the matching debug symbols and rerun the reproducer.

Based on the current evidence, the most logical next steps seem to be:

Code: 
1. Deploy a fresh Vultr instance from the official FreeBSD ISO and run the reproducer.
2. If the failure still occurs, test with virtio_random disabled.
3. If it persists, attempt reproduction on a non-Vultr KVM or bhyve VM.
4. Optionally test the affected host with the default nsswitch.conf as another control.
5. Optionally test in single-user mode to reduce background activity.


So, I would appreciate suggestions on what to test next.

I am willing to run additional tests, provide sanitized logs, rerun the reproducer with suggested changes, or collect more specific diagnostics if that would help. I am intentionally not posting raw core files publicly, but I can keep them available privately if a FreeBSD developer needs them through an appropriate channel.

Thanks in advance for any guidance.

Trapo
 
Hmmm...

So I recall seeing this type of non-deterministic behaviour in the console login program (Alt+Fn). At that time, I think I was on a 14.x on a BETA, and I assumed I was fatfingering my password (keyboard map switch). But on a hunch after turning on core dumps for setuid binaries (sysctl kern.sugid_coredump=1) , I did actually get a login.core file. Which ofcourse I do not have anylonger. This would imply some sort of instability in that code path.

So maybe getting the dbg sysmbols in base installed and having that sysctl turned on should be considered. And then I guess "wait" for core to drop.
 
siladi said:
So maybe getting the dbg sysmbols in base installed and having that sysctl turned on should be considered. And then I guess "wait" for core to drop.
Click to expand...

The debug symbols don't have to be present at coredump time. Only at debugger time. Can even be on a different machine if distribution sets match.
 
OK guys, got a login.core file, 13M. Backtrace below:

Code: 
# gdb -q login
Reading symbols from login...
Reading symbols from /usr/lib/debug//usr/bin/login.debug...
(gdb) core /root/login.core
[New LWP 100369]
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
warning: Can't read pathname for load map.
Core was generated by `login'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x0000098f6e706780 in ?? ()
(gdb) bt
#0  0x0000098f6e706780 in ?? ()
#1  0x0000098f749caa0d in thr_fork_impl (a=0x98f66021a48) at /usr/src/lib/libthr/thread/thr_fork.c:179
#2  0x0000098f749ca988 in __thr_fork () at /usr/src/lib/libthr/thread/thr_fork.c:319
#3  0x0000098745275eaf in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/usr.bin/login/login.c:548
(gdb)

Would be useful if OP or anyone else can get some core files. trapo did you say you got a reproducer ? turn on core dumps for setuid binaries (sysctl kern.sugid_coredump=1) and see if you get something useful.
 
It seems like I might also be affected by this, as my sshd is crashing several times a week, and some other random processes like sudo, which I believe telegraf is spawning frequently. It seems to have started two weeks ago after an emergency maintenance, perhaps there was a hypervisor upgrade or I got moved to different physical hardware or something.

14.4-RELEASE-p7 in their New Jersey location. I can open a support ticket to try and make more noise about this, but I assume it'll take quite a bit for it to get priority.
 
Hi!

I have now completed a more controlled A/B/C comparison, and I think the results are strong enough to narrow this down considerably.

The crashes were repeatedly reproducible on the FreeBSD installation created from Vultr's provided (custom) image, but I could not reproduce them after installing FreeBSD from the official ISO on the same Vultr platform, nor on a separate Hyper-V/Proxmox/QEMU/KVM host using the official ISO.

At this point, I do not think the evidence supports either a generic FreeBSD 15.1-p2 bug or a KVM-only bug.

The failure appears to be strongly associated with the FreeBSD installation produced from Vultr's provided image and/or with the provisioning/customization state of that image.

I still do not know which specific difference is responsible.

Test A — Vultr-provided FreeBSD custom image: FAIL

The original system was FreeBSD 15.1-RELEASE-p2 running on Vultr/KVM.

The first symptom was sshd terminating with SIGSEGV:

Code: 
pid 1723 (sshd), jid 0, uid 0: exited on signal 11

However, I was later able to reproduce the failure locally, without SSH, PAM, a remote client, or external network traffic being involved.

A loop involving:

Code: 
getent passwd root
getent group wheel
id root

caused getent itself to crash with SIGSEGV and generate a core dump.

One backtrace included:

Code: 
strcasecmp_l
_nsdbtput
_nsyyparse
_nsdispatch
setpassent
getent

A second independently reproduced getent crash included:

Code: 
strcasecmp_l
bsearch
_nsdbtaddsrc
_nsyyparse
_nsdispatch
setpassent
getent

This was useful because it showed that, at least in my own case, sshd is not required to trigger the problem.

I also tested whether the stock NSS compat/NIS configuration was necessary.

The original /etc/nsswitch.conf was the stock FreeBSD configuration and included entries such as:

Code: 
group: compat
group_compat: nis
passwd: compat
passwd_compat: nis

I temporarily simplified NSS to files-only sources. The system still eventually reproduced the getent SIGSEGV.

So the compat/NIS entries are not required for the failure.

Locale experiments also did not identify the locale as a necessary condition.

I therefore do not think the evidence currently justifies attributing the crash specifically to sshd, PAM, RSA, locale, NIS, or the compat NSS source.

Test B — same Vultr platform, official FreeBSD ISO: PASS

I then reinstalled the same Vultr instance using the official FreeBSD 15.1-RELEASE amd64 ISO instead of the provider-prepared image.

The clean RELEASE installation was tested first.

It completed approximately:

Code: 
50,000 x getent passwd root

plus

20,000 cycles of:
    getent passwd root
    getent group wheel
    id root

= approximately 110,000 relevant calls

without reproducing the failure.

I then updated this installation through PkgBase to FreeBSD 15.1-RELEASE-p2 and rebooted into p2.

The post-upgrade system completed another approximately 410,000 relevant calls, including five consecutive rounds of 20,000 three-command NSS cycles.

No SIGSEGV or core dump occurred.

The sshd listener also remained stable through the stress tests and subsequent observation period. An important detail is that at 15.1-RELEASE-p2 the critical binaries on the failing Test A installation and the passing Test B installation had identical SHA256 hashes:

Code: 
/lib/libc.so.7
95b6597787b89cec982f44bee5e9af2cf0cd5e5258c1a20ae9cadf8f0421ea5b

/usr/bin/getent
90334c62aa9ba4f2972c99f4691502cfd38ad2114736994e198ec2c1c241a9e9

/usr/bin/id
3104fbf4fc5beccda9c032148dd1f2740d088832c87d099eb8133f3047dfa417

/usr/sbin/sshd
b0f7a9fa64cab1e2f7243bb7fc0ff86f6c115cc2900ba04d1a852dc12b142aa9

So this was not simply a case of Test A running a different libc/getent/id/sshd binary from Test B.

Test C — independent Proxmox/QEMU/KVM host, official ISO: PASS

I wanted an additional control outside Vultr, while still retaining QEMU/KVM and VirtIO.

I therefore created a completely fresh VM on a separate Proxmox host using the official FreeBSD 15.1-RELEASE amd64 ISO.

The relevant virtual hardware was:

Code: 
1 vCPU
1 GiB RAM
SeaBIOS
i440FX/PIIX machine
VirtIO block device
VirtIO network device
KVM
ZFS root

The physical CPU is an Intel Core i5-6500 (Skylake), exposed to the guest with the Proxmox "host" CPU model.

Inside FreeBSD the guest identifies the hypervisor as:

Code: 
Hypervisor: Origin = "KVMKVMKVM"

and the relevant devices as VirtIO block, network, and balloon devices.

I repeated essentially the same sequence as Test B.

First, on stock FreeBSD 15.1-RELEASE:

Code: 
50,000 x getent passwd root

plus

20,000 cycles of:
    getent passwd root
    getent group wheel
    id root

This produced another 110,000 relevant calls without a failure.

I then upgraded the system using the FreeBSD PkgBase repository to 15.1-RELEASE-p2 and rebooted.

After the reboot:

Code: 
freebsd-version -kru

15.1-RELEASE-p2
15.1-RELEASE-p2
15.1-RELEASE-p2

The same four critical files had exactly the same p2 hashes listed above for Tests A and B.

I then repeated the post-p2 stress sequence:

Code: 
50,000 getent passwd calls

60,000 calls in a mixed NSS test

300,000 calls in five additional rounds

for a total of:

Code: 
410,000 post-p2 calls

with no SIGSEGV, no core dump, and no relevant kernel/log error.

Including the pre-p2 tests, Test C has completed approximately:

Code: 
520,000 relevant NSS calls

without reproducing the failure.

During the five-round heavy test, the sshd listener remained the same process throughout all rounds, and the hashes of libc, getent, id, and sshd remained unchanged.

Current A/B/C matrix

Code: 
A: Vultr-provided image
   Vultr + KVM/QEMU/VirtIO + FreeBSD 15.1-p2
   FAIL

B: Official FreeBSD ISO
   Vultr + KVM/QEMU/VirtIO + FreeBSD 15.1-p2
   PASS

C: Official FreeBSD ISO
   Proxmox + QEMU/KVM/VirtIO + FreeBSD 15.1-p2
   PASS

So, I do not think these results support the conclusion that FreeBSD 15.1-RELEASE-p2 itself is generally broken.

They also do not support KVM alone being sufficient to trigger the failure.

The strongest conclusion I think the evidence supports is:


The failure is strongly associated with the FreeBSD installation produced from Vultr's provided image and/or with the provisioning/customization state of that image, rather than with FreeBSD 15.1-p2 or KVM in isolation.
​

This distinction is important. I am not claiming that I have identified a specific Vultr bug or the exact component responsible.

There are still several differences between the original provider-prepared installation and a clean installation from the official ISO, including image/provisioning history, base-system management, loader configuration, filesystem/layout, and potentially other state introduced while the image was being built or deployed.

Because of that, I would not currently attribute the problem specifically to cryptodev, AES-NI, OpenSSL, RSA, VirtIO, sshd, NSS itself, or any other individual component without further evidence.

In particular, the fact that the failing and non-failing p2 systems have identical libc/getent/id/sshd hashes argues against the executable files themselves being sufficient to explain the difference.

It is also worth noting that the original Vultr image had additional loader configuration such as crypto/cryptodev-related modules, while the clean ISO installations did not. However, earlier testing and reports in this thread do not, in my opinion, justify identifying those modules as the root cause.

Practical workaround

For my own systems on Vultr, the practical workaround is now straightforward:


Install FreeBSD from the official FreeBSD ISO instead of using the provider-prepared FreeBSD image.
​

That configuration has so far remained stable under the same reproducer that repeatedly crashed the original image-derived installation.

I have preserved the failing getent core dumps, LLDB backtraces, configuration files, package information, hashes, test logs, and A/B/C manifests.

I have intentionally not posted the raw cores or complete authentication logs here because they may contain information that should not be published publicly, but I can provide appropriately sanitized diagnostic material if it would be useful for a FreeBSD bug report or for further investigation.

At this stage I think a proper bug report may still be useful, especially because the actual mechanism behind the image-dependent behavior remains unidentified. But I would describe the issue there as an image/provisioning-dependent reproducible libc/NSS crash, rather than as a confirmed generic sshd, libc, KVM, or FreeBSD 15.1 bug.

On a more personal note, I have been very happy with my experiences with FreeBSD since 1999, and deeply grateful for the support I have found in this wonderful community. A "Salve" to all of you - and my sincere thanks!

Trapo
 
You must log in or register to reply here.
Back
Top