I'm trying to reach a point where I can do online debugging with kgdb of the FreeBSD kernel and it's modules in order to learn more about the system and take on a small projects in the system development. I have searched through the whole internet for the complete guide on how to do it however found only two that somewhat worked:
1) Kernel Development Recipes BY MARK JOHNSTON
2) Using Bhyve for FreeBSD development by John Baldwin
In this attempt I was following the 1st source in order to create a reusable playground in order to learn and learn the kernel through debugging. I can build and install into a folder, however when I launch bhyve vm using vmrun.sh I can see that the GENERIC config is used from what I can see it is the one in the original image, and the one mounted is ignored somehow. I cannot force the vm to take it first even though it says that it see it at the 'disk0p1:/boot/kernel'. Secondly when I loaded the kernel using 2nd source into bhyve with -H option and load through loader console manually, I could see that the built kernel was loader by the config name but sysctl debug.kdb.available was empty again. Could mean that the built kernel does not have the DDB or KDB included? I can see the *.dbg symbols alongside the source.
Host information:
FreeBSD repository worktree used: origin/stable/14
Steps done:
If anything is missing, please ask, I will add it into the original post.
Help in resolving the setup is greatly appreciated, I searched and read through everything that could, but it just does not click in my mind what is missing. I could use just DTrace for learning, but for the development I think debugging should be more useful, plus seeing how the the dataflow in the kernel and different mecahnisms work online would be more useful.
1) Kernel Development Recipes BY MARK JOHNSTON
2) Using Bhyve for FreeBSD development by John Baldwin
In this attempt I was following the 1st source in order to create a reusable playground in order to learn and learn the kernel through debugging. I can build and install into a folder, however when I launch bhyve vm using vmrun.sh I can see that the GENERIC config is used from what I can see it is the one in the original image, and the one mounted is ignored somehow. I cannot force the vm to take it first even though it says that it see it at the 'disk0p1:/boot/kernel'. Secondly when I loaded the kernel using 2nd source into bhyve with -H option and load through loader console manually, I could see that the built kernel was loader by the config name but sysctl debug.kdb.available was empty again. Could mean that the built kernel does not have the DDB or KDB included? I can see the *.dbg symbols alongside the source.
Host information:
Code:
$ uname -a
FreeBSD t480 14.1-RELEASE-p4 FreeBSD 14.1-RELEASE-p4 GENERIC amd64FreeBSD repository worktree used: origin/stable/14
Code:
~/proj/freebsd-stable/14 $ cat sys/amd64/conf/KERNDEV
ident KERNDEV
include GENERIC
makeoptions DEBUG=”-g -O0”
options KSTACK_PAGES=16
#makeoptions DEBUG="-g -O1" # Build kernel with gdb(1) debug symbols
makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support
# Debugging support. Always need this:
options GDB
options KDB # Enable kernel debugger support.
options DDB
options KDB_TRACE # Print a stack trace for a panic.
options KDB_UNATTENDED
options BREAK_TO_DEBUGGER
#options DDB_UNATTENDED # Don't drop into DDB for a panic
#options GDB_REMOTE_CHAT #Use gdb remote debugging protocol
options KTRACE
options DIAGNOSTIC
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS #Enable checks to detect deadlocks and cycles
options WITNESS_SKIPSPIN #Don't run witness on spinlocks for speed
Code:
$ kldstat
Id Refs Address Size Name
1 97 0xffffffff80200000 1f370e8 kernel
2 1 0xffffffff82138000 5cd608 zfs.ko
3 1 0xffffffff82707000 36c8 coretemp.ko
4 1 0xffffffff8270b000 77d8 cryptodev.ko
5 1 0xffffffff83418000 1b3920 i915kms.ko
6 2 0xffffffff835cc000 7c050 drm.ko
7 1 0xffffffff83649000 22b8 iic.ko
8 3 0xffffffff8364c000 3080 linuxkpi_hdmi.ko
9 3 0xffffffff83650000 6350 dmabuf.ko
10 3 0xffffffff83657000 3378 lindebugfs.ko
11 1 0xffffffff8365b000 b360 ttm.ko
12 1 0xffffffff83667000 3390 acpi_wmi.ko
13 1 0xffffffff8366b000 3250 ichsmb.ko
14 1 0xffffffff8366f000 2178 smbus.ko
15 1 0xffffffff83672000 93260 if_iwlwifi.ko
16 1 0xffffffff83706000 17314 if_iwm.ko
17 1 0xffffffff8371e000 2110 pchtherm.ko
18 1 0xffffffff83721000 1bc6f0 iwm8265fw.ko
19 1 0xffffffff838de000 4d20 ng_ubt.ko
20 3 0xffffffff838e3000 abb8 netgraph.ko
21 2 0xffffffff838ee000 a250 ng_hci.ko
22 2 0xffffffff838f9000 2670 ng_bluetooth.ko
23 1 0xffffffff838fc000 6760 cuse.ko
24 1 0xffffffff83903000 2a68 mac_ntpd.ko
25 1 0xffffffff83906000 b0b0 tmpfs.ko
26 1 0xffffffff83a00000 33e438 vmm.ko
27 1 0xffffffff83912000 7798 if_bridge.ko
28 1 0xffffffff8391a000 60e0 bridgestp.ko
29 1 0xffffffff83921000 21dc nmdm.ko
$ sudo pkg install -y bhyve-firmware
$ export MAKEOBJDIRPREFIX=~/proj/freebsd-stable/obj
$ make -j $(sysctl -n hw.ncpu) buildkernel KMODOWN=user DESTDIR=~/proj/freebsd-stable/kernel KERNCONF=KERNDEV
$ make installkernel KMODOWN=user DESTDIR=~/proj/freebsd-stable/kernel KERNCONF=KERNDEV
$ makefs -B little -S 512 -Z -o version=2 ~/proj/freebsd-stable/kernel/kernfs METALOG
$ mkimg -s gpt -f raw -S 512 -p freebsd-ufs/kern:=~/proj/freebsd-stable/kernel/kernfs -o ~/proj/freebsd-stable/kernel/kernfs.raw
$ sudo sh /usr/share/examples/bhyve/vmrun.sh -E -G localhost:1234 -d ~/proj/freebsd-stable/kernel/kernfs.raw -d ~/proj/FreeBSD-14.1-RELEASE-amd64.raw kerndev
GUEST INFO:
Loader sees:
OK lsdev
disk devices:
disk0: 1677059 X 512 blocks
disk0p1: FreeBSD UFS
disk1: 83886080 X 512 blocks
disk1p1: FreeBSD boot
disk1p2: EFI
disk1p3: FreeBSD swap
disk1p4: FreeBSD UFS
http: (unknown)
net devices:
net0:
OK
BOOTKING PROCESS:
Loading kernel...
disk0p1:/boot/kernel/kernel text=0x17c528 text=0xd69ba8 text=0x42c964 data=0x180+0xe80 data=0x1876e0+0x478920 0x8+0x18ba48+0x8+0x1af4c4\
Loading configured modules...
disk0p1:/boot/kernel/nullfs.ko size 0x97f8 at 0x213b000
/boot/entropy size=0x1000
/etc/hostid size=0x25
staging 0x14000000 (not copying) tramp 0x1e05e000 PT4 0x1e055000
Start @ 0xffffffff8037d000 ...
EFI framebuffer information:
addr, size 0xc0000000, 0x1000000
dimensions 1024 x 768
stride 1024
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-STABLE n268744-79c34d704f31 GENERIC amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(efifb): resolution 1024x768
CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz (1800.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x806ea Family=0x6 Model=0x8e Stepping=10
Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
Features2=0xfeda7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1407a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,SMAP>
Structured Extended Features3=0x400<MD_CLEAR>
XSAVE Features=0x1<XSAVEOPT>
TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory = 536870912 (512 MB)
avail memory = 477577216 (455 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE BVAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-31
Launching APs: 1
random: entropy device external interface
kbd1 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x1fbd0000-0x1fbd001e
smbios0: Version: 2.8, BCD Revision: 2.8
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <BHYVE BVFACP >
acpi0: Power Button (fixed)
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 16777216 Hz quality 950
Event timer "HPET" frequency 16777216 Hz quality 550
Event timer "HPET1" frequency 16777216 Hz quality 450
Event timer "HPET2" frequency 16777216 Hz quality 450
Event timer "HPET3" frequency 16777216 Hz quality 450
Event timer "HPET4" frequency 16777216 Hz quality 450
Event timer "HPET5" frequency 16777216 Hz quality 450
Event timer "HPET6" frequency 16777216 Hz quality 450
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x2100-0x213f mem 0xc1004000-0xc1005fff irq 23 at device 2.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:a0:98:c6:a5:21
vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
000.000185 [ 452] vtnet_netmap_attach vtnet attached txq=1, txd=1024 rxq=1, rxd=512
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0x2080-0x20ff mem 0xc1002000-0xc1003fff irq 16 at device 3.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
vtblk0: 818MB (1677059 512 byte sectors)
virtio_pci2: <VirtIO PCI (legacy) Block adapter> port 0x2000-0x207f mem 0xc1000000-0xc1001fff irq 17 at device 4.0 on pci0
vtblk1: <VirtIO Block Adapter> on virtio_pci2
vtblk1: 40960MB (83886080 512 byte sectors)
vgapci0: <VGA-compatible display> mem 0xc1006000-0xc100607f,0xc0000000-0xc0ffffff at device 29.0 on pci0
vgapci0: Boot video device
vmgenc0: <VM Generation Counter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
Unhandled ps2 mouse command 0xe1
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 15.0.
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x80 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 4 on acpi0
uart3: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff pnpid PNP0900 on isa0
Timecounter "TSC" frequency 1799984873 Hz quality 1000
Timecounters tick every 10.000 msec
usb_needs_explore_all: no devclass
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Dual Console: Serial Primary, Video Secondary
sysctl: debug.kdb.current=gdb at line 13: Invalid argument
Setting hostuuid: e36df0a0-d8cb-de31-813f-50c730617ff7.
Setting hostid: 0x3a3553a7.
Starting file system checks:
/dev/gpt/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/gpt/rootfs: clean, 8991736 free (56 frags, 1123960 blocks, 0.0% fragmentation)
/dev/gpt/efiesp: FILESYSTEM CLEAN; SKIPPING CHECKS
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Setting hostname: freebsd.
Setting up harvesting: PURE_VMGENID,PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
lo0: link state changed to UP
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:c6:a5:21
inet6 fe80::2a0:98ff:fec6:a521%vtnet0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Starting devd.
Starting dhclient.
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 10
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 11
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 5
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
Waiting 30s for the default route interface: .............................
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating /var/run/os-release done.
Clearing /tmp (X related).
Updating motd:.
Creating and/or trimming log files.
Starting syslogd.
Mounting late filesystems:.
Starting cron.
sysctl: debug.kdb.current=gdb at line 13: Invalid argument
Starting background file system checks in 60 seconds.
Fri Sep 20 09:56:27 UTC 2024
FreeBSD/amd64 (freebsd) (ttyu0)
login: root
Last login: Fri Sep 20 09:05:03 on ttyu0
FreeBSD 14.1-STABLE (GENERIC) n268744-79c34d704f31
Sep 20 09:56:32 freebsd login[797]: ROOT LOGIN (root) ON ttyu0
root@freebsd:~ # uname -a
FreeBSD freebsd 14.1-STABLE FreeBSD 14.1-STABLE n268744-79c34d704f31 GENERIC amd64
root@freebsd:~ # cat /etc/sysctl.conf
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
# Fix paths used by the kernel linker
kern.bootfile=/boot/kernel/kernel
kern.module_path=/boot/kernel
debug.kdb.current=gdb
root@freebsd:~ # cat /boot/loader.conf.d/loader.conf
# Make sure that nullfs is available
nullfs_load="YES"
autoboot_delay=2
kernel="disk0p1:/boot/kernel"
kernel_options="-d -g"
root@freebsd:~ # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs / ufs rw 1 1
/dev/gpt/swapfs none swap sw 0 0
/dev/gpt/efiesp /boot/efi msdosfs rw 2 2
# Mount the custom kernel filesystem at /boot/kernel
/dev/gpt/kern /mnt ufs ro 0 0
/mnt/boot/kernel /boot/kernel nullfs ro 0 0
root@freebsd:~ # kldstat
Id Refs Address Size Name
1 3 0xffffffff80200000 1f3af20 kernel
2 1 0xffffffff8213b000 97f8 nullfs.ko
root@freebsd:~ # sysctl debug.kdb
debug.kdb.enter_securelevel: 0
debug.kdb.alt_break_to_debugger: 0
debug.kdb.break_to_debugger: 0
debug.kdb.stack_overflow: 0
debug.kdb.trap_code: 0
debug.kdb.trap: 0
debug.kdb.panic_str:
debug.kdb.panic: 0
debug.kdb.enter: 0
debug.kdb.current:
debug.kdb.available:
root@freebsd:~ #If anything is missing, please ask, I will add it into the original post.
Help in resolving the setup is greatly appreciated, I searched and read through everything that could, but it just does not click in my mind what is missing. I could use just DTrace for learning, but for the development I think debugging should be more useful, plus seeing how the the dataflow in the kernel and different mecahnisms work online would be more useful.
