Unable to trace vtnet_tick() in remote gdb setup

A

ahev

Hello all!

I have set up the following environment:

vmrun.sh -d disk.img -t tap0 -G 1234 thevm

Then I have moved to where I have the kernel compiled and ran gdb from there.
In the VM, I have also run

Code: 
dtrace -n 'fbt:kernel:vtnet_tick:entry'

And I can see that it fires regularly.

The problem is, I cannot break into the function with the use of breakpoints. I set the breakpoint, and that step goes well, but I am unable to stop the execution when I reach that point. Please tell me whether that's a limitation of the setup in general? Should I use the other way to debug the vm, using uart port? Or will that option also not work? Thanks in advance.
 
Not loading the kernel modules required for dtrace does not help. Instead, if I run the virtual machine with the "w" wait option here <-G w1234>, it crashes :( and that happens consistently.
 
bakul said:
Show us your gdb interaction (without using the -w option or dtrace).
Click to expand...
As you said here in the first paragraph (I use the -G option, the one that allows doing Control-C to stop the VM at arbitrary point of time: https://forums.freebsd.org/threads/...etup-for-kernel-development.95026/post-675831), I go to where kernel.full lays, and type
gdb kernel.full
After that, inside gdb, I type
target remote :1234

Side note: it works without "w" option.
With the vm I get the following report:

vm exit[1]
reason VMX
rip 0x0000000000000000
inst_length 0
status 0
exit_reason 33 (VM-entry failure due to invalid guest state)
qualification 0x0000000000000000
inst_type 0
inst_error 0
Abort trap
 
What happens when you try to set the breakpoint on vtnet_tick? Show gdb output as well -- use the script(1) command to make a complete typescript. Or cut-n-paste!
 
Please note that the VM crashes after I type in continue command in gdb.

Terminal One:

# cd /usr/obj/usr/src/FreeBSD/amd64.amd64/sys/GENERIC/
# cat .gdbinit
set sysroot .
# gdb kernel.full
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.full...
warning: File "/usr/obj/usr/src/FreeBSD/amd64.amd64/sys/GENERIC/.gdbinit" auto-loading has been declined by your `au
to-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /usr/obj/usr/src/FreeBSD/amd64.amd64/sys/GENERIC/.gdbinit
line to your configuration file "/root/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) target remote :1234
could not connect: Operation timed out.
(gdb) target remote :1234
Remote debugging using :1234
0xffffffff80389000 in cam_compat_handle_0x17 (dev=0xffffffffffffffff, cmd=18446744073709551615, addr=0x0, flag=-1,
td=0xffffffffffffffff, cbfnp=0xffffffffffffffff) at /usr/src/FreeBSD/sys/cam/cam_compat.c:180
180 bcopy(&cts17->xport_specific, &cts->xport_specific,
(gdb) continue
Continuing.
Remote connection closed
(gdb) quit
# ^D

Terminal Two:

______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s` `.....---.......--.``` -/
+---------- Welcome to FreeBSD -----------+ +o .--` /y:` +.
| | yo`:. :o `+-
| 1. Boot Multi user [Enter] | y/ -/` -o/
| 2. Boot Single user | .- ::/sy+:.
| 3. Escape to loader prompt | / `-- /
| 4. Reboot | `: :`
| 5. Cons: Serial | `: :`
| | / /
| Options: | .- -.
| 6. Kernel: default/kernel (1 of 1) | -- -.
| 7. Boot Options | `:` `:`
| | .-- `--.
| Loader needs to be updated | .---.....----.
| |
+-----------------------------------------+
Autoboot in 7 seconds. [Space] to pause
Loading kernel...
/boot/kernel/kernel text=0x187fc8 text=0xdc6928 text=0x487f64 data=0x180+0xe80 data=0x1ada00+0x452600 0x8+0x1a3e08+0
x8+0x1cfea9\
Loading configured modules...
/boot/entropy size=0x1000
/etc/hostid size=0x25
vm exit[1]
reason VMX
rip 0x0000000000000000
inst_length 0
status 0
exit_reason 33 (VM-entry failure due to invalid guest state)
qualification 0x0000000000000000
inst_type 0
inst_error 0
Abort trap
artem@hevorhian:~ $ ^D
 
Also, I have the following interesting behavior on my machine:
artem@hevorhian:~ $ sudo bhyvectl --vm=thevm --destroy
artem@hevorhian:~ $ sudo bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,vm0/disk.img \
-l com1,stdio -A -H -P -m 1G thevm
virtual machine cannot be booted
artem@hevorhian:~ $ sudo bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,vm0/disk.img \
-l com1,stdio -A -H -P -m 1G thevm
vm exit[0]
reason VMX
rip 0x0000000000000000
inst_length 0
status 0
exit_reason 33 (VM-entry failure due to invalid guest state)
qualification 0x0000000000000000
inst_type 0
inst_error 0
Abort trap
 
(kgdb) b vtnet_tick
Note: breakpoints 1, 2 and 3 also set at pc 0xffffffff8097dba4.
Breakpoint 4 at 0xffffffff8097dba4: file /usr/src/FreeBSD/sys/dev/virtio/network/if_vtnet.c, line 3048.
(kgdb) cont
Continuing.


I have figured out how to launch bhyve into memory, for BIOS based images I need to use bhyveload before I run bhyve. But there's this problem that I have faced with just running vmrun.sh script. I cannot break into vtnet_tick function. Here's how I launched bhyve

sudo bhyve \
-H -A -P \
-c 8 \
-m 8G \
-G 1234 \
-s 0,hostbridge \
-s 1,virtio-blk,./vm0/disk.img \
-2 2,vrtio-net,tap0
-s 31,lpc \
-l com1,stdio \
thevm

Now I can't see neither in gdb nor in dtrace vtnet_tick() at all. Strange.. I would suppose that if I use virtio-net backend, the driver should work.
 
Maybe I should debug in this second way: https://forums.freebsd.org/threads/...tup-for-kernel-development.95026/post-675831? Because the first (with -G option using the gdb stub in bhyve) does not work? I would like to be able to set arbitrary breakpoints in the kernel, so that I can debug drivers, for example.

Edit: I have tried debugging using serial console (nmdm devices) and the result is absolutely the same.

I also found the following error when debugging using -G option:
Error return from kevent change: Bad file descriptor
 
I have found the problem. For vtnet_tick() to fire, one needs to assign an IP address to the vtnet0 interface. And the regular -G in-bhyve gdb stub works perfectly with it, as with everything else.
 
Don't forget to put 'set sysroot .' in .gdbinit in the same directory as kernel.full or else it will lookup symbols from /usr/lib/debug/boot/kernel for your host (which may be a different kernel)! In addition 'ln -s **/*.ko .' (in zsh) -- basically symlink all .ko files to appear in the same dir as kernel.full so that kgdb or gdb can find all the symbols.
 
You must log in or register to reply here.
Back
Top