FreeBSD 13.1 in KVM VM: Regular panics with various messages

Background and Setup

I've got a setup with 2 machines. Both are random AliExpress specials ("Techvision") Celeron N5105 CPUs and quad 2.5GB i225-V Intel NICs, a 128GB NVMe SSD, and 8GB of RAM. On the bare metal is Debian with QEMU/KVM + Libvirt, and then on each machine I run a virtual guest with FreeBSD 13.1.

Into these guests I directly pass 3 of the 4 Intel NICs. The purpose of these machines are to act as routers for my network with numerous vLANs.

On the FreeBSD system I did have to compile the i225 driver ("igc") myself using ports ("cd /usr/src/sys/modules/igc", "sudo make" copy into "/boot/modules" and "/boot/kernel"), but it's completely stock/unchanged otherwise.

The Problem

The virtual FreeBSD guests are kernel panic'ing up to once every 2-3 days. It's happening to both guests at random times. The crashes are also seemingly random, with different messages for each. A few examples are below. It also sometimes just freezes and drops networking, without a panic/restart.

Troubleshooting I've Done

I've done a full memtest on both machines and it comes back clean. I've also reinstalled one VM with the latest 13.1 release, while the other was originally 13.0 and upgraded, to no effect. One is ZFS root while the other is UFS root, no change.

The Request

I have no clue what's going on here, especially given that it's not the same error each time, and my rudimentary FreeBSD skills are failing me here. I'm hoping someone might have some idea why this is occurring and give some potential ideas for workarounds I could try. And if it's a bug I'd like to report it, but that is daunting to me. I do have kernel dumps of these last few crashes but not sure how to easily share the 500MB files (please let me know).

Example Crashes

Here are the 4 most recent crashes. As can be seen, each one is different, though they tend to be one of the two types shown "Fatal double fault" or "Page fault while in kernel mode".

Code:
Fatal double fault
rip 0xffffffff81073826 rsp 0xfffffe000b991dd0 rbp 0xfffffe000b991dd0
rax 0x176a0a152f21a0 rdx 0x176a0a00000000 rbx 0xfffff80003906800
rcx 0 rsi 0 rdi 0xfffff8000319a388
r8 0x350 r9 0xfffff8000319a000 r10 0xfffff8000319a390
r11 0x1 r12 0xfffff80003906828 r13 0x3
r14 0x176a0a152f21a0 r15 0x12 rflags 0x10246
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x3e4076137150 gsbase 0xffffffff82a11000 kgsbase 0
cpuid = 1; apic id = 01

Fatal double fault
rip 0xffffffff81073826 rsp 0xfffffe000b987dd0 rbp 0xfffffe000b987dd0
rax 0x176a0a1511e618 rdx 0x176a0a00000000 rbx 0xfffff80003906000
rcx 0 rsi 0 rdi 0xfffff8000319aa28
r8 0x9f0 r9 0xfffff8000319a000 r10 0xfffff8000319aa30
r11 0x1 r12 0xfffff80003906028 r13 0
r14 0x176a0a1511e618 r15 0xb rflags 0x10246
cs 0x20 ss 0x28 ds 0x3b es 0x3bpanic: double fault
cpuid = 1
time = 1684656930
KDB: stack backtrace:
Uptime: 1d4h48m9s
Dumping 475 out of 6102 MB:..4%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1

Code:
kernel trap 1 with interrupts disabled

Fatal trap 1: privileged instruction fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff8117d872
stack pointer           = 0x0:0xfffffe00bd782f38
frame pointer           = 0x0:0x222723e50f20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 73050 (sed)
trap number             = 1
panic: privileged instruction fault
cpuid = 0
time = 1682705937
KDB: stack backtrace:
#0 0xffffffff80c53dc5 at kdb_backtrace+0x65
#1 0xffffffff80c06741 at vpanic+0x151
#2 0xffffffff80c065e3 at panic+0x43
#3 0xffffffff810b1fa7 at trap_fatal+0x387
#4 0xffffffff81088e78 at calltrap+0x8

Code:
kernel trap 1 with interrupts disabled


Fatal trap 1: privileged instruction fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff8117d872
stack pointer           = 0x0:0xfffffe00bd782f38
frame pointer           = 0x0:0x222723e50f20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 73050 (sed)
trap number             = 1
panic: privileged instruction fault
cpuid = 0
time = 1682705937
KDB: stack backtrace:
#0 0xffffffff80c53dc5 at kdb_backtrace+0x65
#1 0xffffffff80c06741 at vpanic+0x151
#2 0xffffffff80c065e3 at panic+0x43
#3 0xffffffff810b1fa7 at trap_fatal+0x387
#4 0xffffffff81088e78 at calltrap+0x8

Code:
Fatal double fault
rip 0xffffffff81073826 rsp 0xfffffe000b991dd0 rbp 0xfffffe000b991dd0
rax 0x1de8809ba098 rdx 0x1de800000000 rbx 0xfffff80003900800
rcx 0 rsi 0 rdi 0xfffff8000319a388
r8 0x350 r9 0xfffff8000319a000 r10 0xfffff8000319a390
r11 0x1 r12 0xfffff80003900828 r13 0x3
r14 0x1de8809ba098 r15 0x7 rflags 0x10246
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x8223d0080 gsbase 0xffffffff82811000 kgsbase 0
cpuid = 1; apic id = 01
timeout stopping cpus
panic: double fault
cpuid = 1
time = 1681372673
KDB: stack backtrace:

Fatal double fault
rip 0xfUptime: 4h13m23s
fffffff81088ef0 rsp 0xffffffff81d530d0 rbp 0xfffffe000b9b9360
Dumping 466 out of 6104 MB:rax 0x201 rdx 0xfffffe000b9b94f0 rbx 0xfffff80068961700
rcx 0xe rsi 0xfffff80003ce2800 rdi 0xfffff80068961700
r8 0x855a707b4be60 r9 0xfffffe000b9b92fc r10 0xfffffe000b9b94f0
r11 0x4 r12 0xfffffe000b9b94d0 r13 0xe
r14 0xfffffe000b9b9320 r15 0xfffff80003ce2800 rflags 0x10086
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x82171b120 gsbase 0xffffffff82810000 kgsbase 0
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
time = 1681372673
KDB: stack backtrace:
Uptime: 4h13m23s

Libvirt VM Definition

Here's the VM definition, just in case it's relevant.

Code:
<domain type='kvm' id='6'>                                                                                          
  <name>dcr1</name>            
  <uuid>1c651633-637f-4a56-a01a-15dee6ee1943</uuid>
  <description>Datacenter Router</description>
  <memory unit='KiB'>6291456</memory>  
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>                                          
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
    <bios useserial='yes' rebootTimeout='5'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <topology sockets='1' dies='1' cores='4' threads='1'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>                                                                               
      <driver name='qemu' type='raw' discard='unmap'/>
      <source dev='/dev/vg/dcr1_sda' index='1'/>
      <backingStore/>                                                                                               
      <target dev='sda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </disk>                                                                                                         
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>                               
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>                                                                                         
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/0'/>                      
      <log file='/var/log/libvirt/dcr1.log' append='on'/>                                                           
      <target type='isa-serial' port='0'>                                                                           
        <model name='isa-serial'/>
      </target>                                   
      <alias name='serial0'/>
    </serial>                          
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <log file='/var/log/libvirt/dcr1.log' append='on'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <rate bytes='2048' period='1000'/>
      <backend model='random'>/dev/random</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-1c651633-637f-4a56-a01a-15dee6ee1943</label>
    <imagelabel>libvirt-1c651633-637f-4a56-a01a-15dee6ee1943</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+64055</label>
    <imagelabel>+64055:+64055</imagelabel>
  </seclabel>
</domain>

rc.conf

Here's the initial parts of my rc.conf; I've excluded all my vLAN configs because they're PII (public IPs and network design aspects) but hopefully that isn't too big of a deal.

Code:
hostname="dcr1.m.bonilan.net"                    
clear_tmp_enable="YES"                           
local_unbound_enable="YES"                                 
sshd_enable="YES"                                
sshguard_enable="YES"                            
ntpd_enable="YES"                                
powerd_enable="YES"                              
dumpdev="AUTO"                                             
zfs_enable="YES"              
devd_enable="YES"                         
inetd_enable="YES"
openbgpd_enable="YES"

# Ifstated configuration
ifstated_enable="YES"
ifstated_profiles="wan vpn "
ifstated_wan_configfile="/usr/local/etc/ifstated.conf-wan"
ifstated_vpn_configfile="/usr/local/etc/ifstated.conf-vpn"

# Wireguard site-to-site (managed via devd)
wireguard_enable="NO"
wireguard_interfaces="vpn1 vpn2 vpn3 "

# DHCPD
dhcpd_enable="YES"
dhcpd_conf="/usr/local/etc/dhcpd.conf"
dhcpd_ifaces="vlan20 vlan40 vlan41 vlan42 vlan62 vlan100 vlan110 "
dhcpd_chuser_enable="YES"
dhcpd_withuser="dhcpd"
dhcpd_withgroup="dhcpd"

# PF
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pfsync_enable="YES"
pfsync_syncdev="igc0"

# Cloned interfaces
cloned_interfaces="lagg0 vlan11 vlan12 vlan13 vlan20 vlan40 vlan41 vlan42 vlan60 vlan62 vlan80 vlan90 vlan100 vlan110 " 

# LACP
ifconfig_igc1="up -tso4 -tso6 -lro -vlanhwtso"
ifconfig_igc2="up -tso4 -tso6 -lro -vlanhwtso"
ifconfig_lagg0="laggproto lacp lagghash l2,l3 laggport igc1 laggport igc2"

# CARP interface
ifconfig_igc0="up -tso4 -tso6 -lro -vlanhwtso inet 10.0.0.1/29"

# EXCLUDED VLANS FOLLOW
 
There was/is a huge thread on servethehome.com forums about those units (primarily sold under the topton label), where multiple people have/had problems with linux-based hypervisors (proxmox etc) and randomly crashing/panicking guests.
IIRC it boiled down to some KVM bug related to those newer CPUs/SoCs, which triggered much more frequently with FreeBSD than other guests (but other OS as well). So if you really have to use linux on the host, you may have to use something that uses a more recent kernel and/or KVM version than debian.

I ran FreeBSD/bhyve with some small VMs on one of my topton units for some small tests for a few weeks and this worked flawlessly. All those units are now running bare metal OpenBSD though, as they are used for what they were designed for: home- and/or VPN routers.
 
Driver is included with the GENERIC kernel. No need to compile it separately.


Curious it wasn't there on my fresh 13.1 guest, though perhaps I did something wrong during install. I'll probably try again soon.

There was/is a huge thread on servethehome.com forums about those units (primarily sold under the topton label), where multiple people have/had problems with linux-based hypervisors (proxmox etc) and randomly crashing/panicking guests.
IIRC it boiled down to some KVM bug related to those newer CPUs/SoCs, which triggered much more frequently with FreeBSD than other guests (but other OS as well). So if you really have to use linux on the host, you may have to use something that uses a more recent kernel and/or KVM version than debian.

I ran FreeBSD/bhyve with some small VMs on one of my topton units for some small tests for a few weeks and this worked flawlessly. All those units are now running bare metal OpenBSD though, as they are used for what they were designed for: home- and/or VPN routers.

That's discouraging, though I can certainly try the latest kernel on one of them (upgrade to Debian 12 instead of 11) and see if it improves things.
 
In case it is helpful, this could be another potential culprit, and it has nothing to do with FreeBSD: https://bugzilla.kernel.org/show_bug.cgi?id=199727

It's a open bug reported 5 years ago regarding QEMU/KVM, where heavily loaded VM's freeze randomly. It's an interesting read, but essentially it boils down to this...

QEMU/KVM on slow hardware causes random lockups when VM's are busy due to the QEMU global mutex. Moving away from virtio-blk drivers to virtio-scsi drivers, implementing iothreads and disabling drive cache, appears to fix the issue for many users.


I am using QEMU with NVMM on DragonFlyBSD, and experienced frequent lockups on an under powered local machine (10 year old MacMini). However, a small Xeon dedicated machine only experienced it a few times, and a Dual CPU 20 core machine with 128G RAM and SSD's has never experienced any lockups. Other than the hardware, everything is identical between all three machines.

After switching to virtio-scsi, using the iothread directive and disabling drive cache, there hasn't been an issue. So, you might keep that in mind, in case a newer kernel doesn't do the trick...
 
Funny enough I already do those 3 things just as a standard for performance, but that's very good to keep in mind.

It does seem, so far, like the upgrade of the hypervisors to Bookworm has solved it. I'm up to nearly a week on the host I upgraded first with no crashes (knock on wood). That's exactly the sort of thing I was hoping to find with this thread so thanks everyone!
 
Those dumps you shared are interesting. The state of the dump shows .. mhm, garbage.

cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
%ds/%es/%fs/%gs are junk and don't point to proper selectors.

stack pointer = 0x0:0xfffffe00bd782f38 frame pointer = 0x0:0x222723e50f20
%ss selector is NULL.

Output where you have the stack trace would be interesting but it seems short. Is that all you saw ?

It almost seems that the AliExpress machines have some serious HW problems. As if information you stored in memory on host just got wildly overwritten.
 
Back
Top