16.0-CURRENT: how to debug a core file from kernel and debug symbols downloaded via pkgbase?

C

ccfbsd

Developer
The kernel in question is a downloaded version via pkgbase.

But it looks kgdb does not fully recognize it. How to proceed? I think the problem is "warning: 399 /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: No such file or directory"

cc@macfbsd16e:/var/crash % uname -a
FreeBSD macfbsd16e 16.0-CURRENT FreeBSD 16.0-CURRENT main-n284009-e387d9438ba0 GENERIC arm64
cc@macfbsd16e:/var/crash % sudo cat info.last
Dump header from device: /dev/nda0p3
Architecture: aarch64
Architecture Version: 1
Dump Length: 152383488
Blocksize: 512
Compression: none
Dumptime: 2026-02-20 09:02:55 -0500
Hostname: macfbsd16e
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 16.0-CURRENT main-n284009-e387d9438ba0 GENERIC
Panic String: Assertion cur != QIDX_INVALID failed at /home/pkgbuild/worktrees/main/sys/dev/e1000/em_txrx.c:499
Dump Parity: 2269964289
Bounds: 1
Dump Status: good
cc@macfbsd16e:/var/crash % sudo kgdb /usr/lib/debug/boot/kernel/kernel.debug /var/crash/vmcore.last
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-portbld-freebsd16.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/debug/boot/kernel/kernel.debug...
Reading symbols from /boot/kernel/hms.ko...
Reading symbols from /usr/lib/debug//boot/kernel/hms.ko.debug...
Reading symbols from /boot/kernel/hidmap.ko...
Reading symbols from /usr/lib/debug//boot/kernel/hidmap.ko.debug...
Reading symbols from /boot/kernel/mac_ntpd.ko...
Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...
0xffff0000004d6a60 in doadump (textdump=0) at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:399
warning: 399 /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: No such file or directory
(kgdb) bt
#0 0xffff0000004d6a60 in doadump (textdump=0) at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:399
#1 0x5a900000000ec9f8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(kgdb)
Click to expand...
 
I may be mistaken, but it looks like the warning about /home/pkgbuild/worktrees/main/... is expected when debugging a pkgbase kernel. The debug symbols embed the original build path from the package build host, which does not exist on the target system.

If you have the corresponding source tree locally (matching commit e387d9438ba0), you can map the build path inside kgdb using:

set substitute-path /home/pkgbuild/worktrees/main /usr/src

After that, bt and list should resolve the source files correctly.

Make sure your /usr/src is checked out at the exact revision reported by uname -a (main-n284009-e387d9438ba0), otherwise line numbers may not align with the debug info.

Also note that the panic itself is in sys/dev/e1000/em_txrx.c:499, so this appears to be an assertion in the Intel e1000 driver rather than a kgdb/debug symbol issue. Once the source path is mapped correctly, you should be able to inspect the failing condition directly in em_txrx.c.

If I'm overlooking something, please feel free to correct me.
 
Thanks.

The source tree was downloaded together with the kernel as I frequently upgrade the base via pkgbase in 16-CURRENT.
pkg upgrade -r FreeBSD-base
Click to expand...

However, the "set substitute-path" looks to be no-op this time.
(kgdb) set substitute-path /home/pkgbuild/worktrees/main /usr/src
(kgdb) bt
#0 0xffff0000004d6a60 in doadump (textdump=0) at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:399
#1 0x5a900000000ec9f8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(kgdb)
Click to expand...
 
It looks like the substitute-path command itself is accepted, but the backtrace still stops immediately. The “previous frame identical to this frame (corrupt stack?)” message suggests this may not be a source mapping issue, but rather a stack unwind problem.

Since the panic is reported in sys/dev/e1000/em_txrx.c:499 but the current frame is doadump(), the system is already inside the panic handler. If the stack was corrupted before or during panic processing, kgdb may not be able to unwind further.

You might try examining frame 1 explicitly, or checking registers (info registers) to see whether unwinding is possible. On arm64 especially, missing or inconsistent frame information can result in a truncated backtrace.
 
Thanks a lot!

Looks the VM has experienced multiple crashes that the file system was not clean before this core dump. I will look for recovery or a fresh new VM.
 
You must log in or register to reply here.
Back
Top