PF Kernel instability

arabesc · Tuesday at 10:20 AM

Hello everyone,

I am currently running FreeBSD 14.3 with the PF firewall on my home server, which acts as a file server, an internet gateway, and hosts several other services. My system has evolved over the course of more than 10 years, starting back from version 10.0. For the most part, it is very stable and trouble-free. The uptime seems like it could be infinite, provided I don't touch anything.

A couple of days ago, I decided to experiment with traffic prioritization. The goal was to reduce the impact of intensive torrent traffic generated by Transmission. I enabled several options in the kernel configuration:

Code:

options        ALTQ
options        ALTQ_PRIQ
options        ALTQ_HFSC
options        ALTQ_CODEL

I configured the queues in PF, enabled them, and immediately encountered kernel panics

. After booting, the system stays alive for about 20 seconds before crashing. If I manage to disable Transmission within that timeframe, the system survives longer. Transmission generates intensive UDP traffic, specifically a high volume of small packets.
I cannot state this with certainty, but I believe that using tags and virtual network interfaces in the PF rules might also contribute to the instability. Without any of these features enabled, my system is stable; as soon as I turn any of them on, the kernel crashes quickly.

I would like to ask if these are known issues, and if the aforementioned technologies (ALTQ, tags, VIFs with PF) are indeed unstable or not ready for production use?

cracauer@ · Tuesday at 10:34 AM

"Crashing" means what, exactly?

mer · Tuesday at 11:14 AM

What happens if you disable all the queueing in PF and start with a reboot? Basically having the options in the kernel, but not actually being used anywhere?
Is the system stable?

arabesc · Tuesday at 11:32 AM

cracauer@ said:
"Crashing" means what, exactly?

Code:

Dec  9 03:27:56 kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 03:27:56 kernel: cpuid = 2; apic id = 02
Dec  9 03:27:56 kernel:
Dec  9 03:27:56 kernel: fault virtual address       = 0x20
Dec  9 03:27:56 kernel: fault code          = supervisor read data, page not present
Dec  9 03:27:56 kernel: instruction pointer = 0x20:0xffffffff82439675
Dec  9 03:27:56 kernel: stack pointer               = 0x28:0xfffffe001b184360
Dec  9 03:27:56 kernel: frame pointer               = 0x28:0xfffffe001b184400
Dec  9 03:27:56 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b
Dec  9 03:27:56 kernel:                     = DPL 0, pres 1, long 1, def32 0, gran 1
Dec  9 03:27:56 kernel:
Dec  9 03:27:56 kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 03:27:56 kernel: cpuid = 4; apic id = 04
Dec  9 03:27:56 kernel: fault virtual address       = 0x20
Dec  9 03:27:56 kernel: fault code          = supervisor read data, page not present
Dec  9 03:27:56 kernel:
Dec  9 03:27:56 kernel: instruction pointer = 0x20:0xffffffff82439675
Dec  9 03:27:56 kernel: stack pointer               = 0x28:0xfffffe001b17a360
Dec  9 03:27:56 kernel: frame pointer               = 0x28:0xfffffe001b17a400
Dec  9 03:27:56 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b
Dec  9 03:27:56 kernel: processor eflags    = interrupt enabled, resume, IOPL = 0
Dec  9 03:27:56 kernel: current process             = 0 (if_io_tqg_2)
Dec  9 03:27:56 kernel: rdi: fffffe001b184968 rsi: 0000000000005c31 rdx: 0000000000000008
Dec  9 03:27:56 kernel:
Dec  9 03:27:56 kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 03:27:57 kernel:                     = DPL 0, pres 1, long 1, def32 0, gran 1
Dec  9 03:27:57 kernel: cpuid = 6; apic id = 06
Dec  9 03:27:57 kernel: fault virtual address       = 0x20
Dec  9 03:27:57 kernel: fault code          = supervisor read data, page not present
Dec  9 03:27:57 kernel: instruction pointer = 0x20:0xffffffff82439675
Dec  9 03:27:57 kernel: stack pointer               = 0x28:0xfffffe001b198360
Dec  9 03:27:57 kernel: frame pointer               = 0x28:0xfffffe001b198400
Dec  9 03:27:57 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b
Dec  9 03:27:57 kernel:                     = DPL 0, pres 1, long 1, def32 0, gran 1
Dec  9 03:27:57 kernel: processor eflags    = interrupt enabled, resume, IOPL = 0
Dec  9 03:27:57 kernel: current process             = 0 (if_io_tqg_4)
Dec  9 03:27:57 kernel: rcx: 0000000000000000  r8: 0000000000000000  r9: 00000000822200aa
Dec  9 03:27:57 kernel: rax: 0000000000009062 rbx: 0000000000000002 rbp: fffffe001b184400
Dec  9 03:27:57 kernel:
Dec  9 03:27:57 kernel: r10: fffffe001b184802 r11: 0000000000009062 r12: fffff801ddc39700
Dec  9 03:27:57 kernel: current process             = 0 (if_io_tqg_6)
Dec  9 03:27:57 kernel: rdi: fffffe001b198968 rsi: 000000000000caac rdx: 0000000000000008
Dec  9 03:27:57 kernel:
Dec  9 03:27:57 kernel: Fatal trap 12: page fault while in kernel mode

Code:

Dec  9 03:53:02 sun kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 03:53:02 kernel: cpuid = 7; apic id = 07
Dec  9 03:53:02 kernel: fault virtual address       = 0x22
Dec  9 03:53:02 kernel: fault code          = supervisor read data, page not present
Dec  9 03:53:02 kernel: instruction pointer = 0x20:0xffffffff82439675
Dec  9 03:53:02 kernel: stack pointer               = 0x28:0xfffffe01090f5250
Dec  9 03:53:02 kernel: frame pointer               = 0x28:0xfffffe01090f52f0
Dec  9 03:53:02 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b
Dec  9 03:53:02 kernel:                     = DPL 0, pres 1, long 1, def32 0, gran 1
Dec  9 03:53:02 kernel: processor eflags    =
Dec  9 03:53:02 kernel: interrupt enabled,
Dec  9 03:53:02 kernel: resume, IOPL = 0
Dec  9 03:53:02 kernel: current process             = 1393 (transmission-daemon)

Code:

Dec  9 04:43:43 kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 04:43:43 kernel: cpuid = 7; apic id = 07
Dec  9 04:43:43 kernel: fault virtual address       = 0x22
Dec  9 04:43:43 kernel: fault code          = supervisor read data, page not present
Dec  9 04:43:43 kernel: instruction pointer = 0x20:0xffffffff824393f5
Dec  9 04:43:43 kernel: stack pointer               = 0x28:0xfffffe0111321260
Dec  9 04:43:43 kernel: frame pointer               = 0x28:0xfffffe0111321300
Dec  9 04:43:43 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b
Dec  9 04:43:43 kernel:                     = DPL 0, pres 1, long 1, def32 0, gran 1
Dec  9 04:43:43 kernel: processor eflags    = interrupt enabled, resume, IOPL = 0
Dec  9 04:43:43 kernel: current process             = 2096 (transmission-daemon)

Code:

Dec  9 14:15:09 kernel: Fatal trap 12: page fault while in kernel mode
Dec  9 14:15:09 kernel: cpuid = 4; apic id = 04
Dec  9 14:15:09 kernel: fault virtual address       = 0x20
Dec  9 14:15:09 kernel: fault code          = supervisor read data, page not present
Dec  9 14:15:09 kernel: instruction pointer = 0x20:0xffffffff824393f5
Dec  9 14:15:09 kernel: stack pointer               = 0x28:0xfffffe001b17a370
Dec  9 14:15:09 kernel: frame pointer               = 0x28:0xfffffe001b17a410
Dec  9 14:15:09 kernel: code segment                = base 0x0, limit 0xfffff, type 0x1b

and then the system reboots.

arabesc · Tuesday at 11:38 AM

mer said:
What happens if you disable all the queueing in PF and start with a reboot? Basically having the options in the kernel, but not actually being used anywhere?
Is the system stable?

The system was absolutely stable until the changes started. Now I'm trying to roll everything back to the initial state, and the uptime has grown from seconds to hours, but the system is still unstable.

For example, one of the remaining changes that I added:

Code:

ifconfig_lo0_alias0="inet 127.0.0.2 netmask 255.0.0.0"

and I run Transmission on this address (to be able to filter its packets by IP); previously I ran it on 127.0.0.1.

cracauer@ · Tuesday at 11:39 AM

Do you have enough swapspace to save a core file?

arabesc · Tuesday at 2:54 PM

cracauer@ said:
Do you have enough swapspace to save a core file?

I suddenly found that my swap was turned off, because the system disk it is located on was renamed at some point from /dev/nvd0
to /dev/diskid/DISK-xxxxxxxxxxxxxxx. I turned the swap back on. The swap size is equal to the RAM size; I don't know if that will be enough for the dump. I will monitor how the system works.

atax1a · Tuesday at 5:09 PM

you can avoid the problems with device names by using a GPT partitioning scheme and glabel, fwiw.

our usual recommendation when hearing about random somewhat-consistent crashes is (broken record voice) is the machine using ECC ram? if not, have you done a memtest?

CeXP1917 · Tuesday at 5:21 PM

arabesc said:
The goal was to reduce the impact of intensive torrent traffic generated by Transmission.

PF can do this with dnqueue/dnpipe.

arabesc · Tuesday at 6:21 PM

atax1a said:
is the machine using ECC ram?

Yep, it's running ECC memory. A hardware fault is still possible, but it seems unlikely as the issues started right after the config changes.

arabesc · Tuesday at 7:17 PM

cracauer@ said:
Do you have enough swapspace to save a core file?

Here is a ~~core dump~~ (~ 299 MB).
upd: I removed the link due to security concerns. Please leave a message if you need a copy, I'll PM the link.

_martin · Tuesday at 8:16 PM

arabesc Please beware kernel dump includes information you may not want to share to public

You built your custom kernel. This means this particular crashdump is linked to your kernel. We can't use generic kernel with this dump. Can you upload /usr/lib/debug/boot/kernel/kernel.debug and possible tar the whole /boot/kernel directory and upload that to dropbox too?

From the dump you shared this stack trace is interesting:

Code:

Timeout initializing vt_vga
panic: page fault
cpuid = 4
time = 1765306293
KDB: stack backtrace:
#0 0xffffffff8073db6d at kdb_backtrace+0x5d
#1 0xffffffff806eeeef at vpanic+0x15f
#2 0xffffffff806eed83 at panic+0x43
#3 0xffffffff80a35bfa at trap_pfault+0x3da
#4 0xffffffff80a0d688 at calltrap+0x8
#5 0xffffffff82435b60 at pf_test_rule+0x2030
#6 0xffffffff82430d83 at pf_test+0x1cf3
#7 0xffffffff8244fba7 at pf_check_in+0x27
#8 0xffffffff8083a038 at pfil_mbuf_in+0x38
#9 0xffffffff8086af29 at ip_input+0x389
#10 0xffffffff80836faf at netisr_dispatch_src+0x9f
#11 0xffffffff8081b39a at ether_demux+0x12a
#12 0xffffffff8081c6a6 at ether_nh_input+0x366
#13 0xffffffff80836faf at netisr_dispatch_src+0x9f
#14 0xffffffff8081b6f6 at ether_input+0x56
#15 0xffffffff80833666 at iflib_rxeof+0xc36
#16 0xffffffff8082e032 at _task_fn_rx+0x72
#17 0xffffffff8073c4ee at gtaskqueue_run_locked+0x14e

I'd be interesting to see more into frame #5.

arabesc · Tuesday at 8:33 PM

_martin said:
Can you upload /usr/lib/debug/boot/kernel/kernel.debug

Here it is.

_martin said:
and possible tar the whole /boot/kernel directory and upload that to dropbox too?

And here it is.

_martin · Wednesday at 8:59 PM

I was not able to replicate the crash using your custom kernel. I see bits of pf.conf but not all of it. Would it be possible to share it? I'll try to dummy fit that to my setup. You can change your public IP to 192.2.0.1 or similar.

Crash is happening in pf.ko module, in pf_return function. One of the structure is null, hence the error.