bhyve ifconfig needs >16s, mainly sbwait on FreeBSD 14.2-p1

G

gtz42

Hi,
I'm running a minimal FreeBSD 14.2-p1/-p3 setup on an Intel ATOM C3858 (12 core), Supermicro A2SDi-TP8F.
The system has 8 onboard nice (4 igb (Quadport I350), 4 ix (two dual port x553),
I've installed bhyve with one VM, which gets 4 NICs passthrough (2x igb, 2x ix).
As far as I can tell, everything works as expected, except that `ifconfig` takes >16s, most of it spend in sbwait:

Code: 
root@box:~ # ifconfig
load: 0.03  cmd: ifconfig 23246 [sbwait] 1.44r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 1.84r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 2.48r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 2.89r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.03r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.25r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.42r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.60r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.77r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 4.93r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 5.10r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 5.25r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 5.45r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 5.67r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 5.84r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.03r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.22r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.44r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.61r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.77r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 6.96r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 7.16r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 7.35r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 7.54r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 7.75r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 7.97r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 8.14r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 8.33r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 8.50r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 8.67r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 8.86r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 9.05r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 9.22r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 9.44r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 9.62r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 9.80r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.04r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.24r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.44r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.62r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.79r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 10.98r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 11.17r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 11.39r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 11.58r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 11.77r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 11.94r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 12.16r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 12.35r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 12.54r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 12.75r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 12.97r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 13.14r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 13.36r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 13.56r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 13.74r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 14.15r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 14.35r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 14.56r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 14.76r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 14.98r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 15.21r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 15.43r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 15.64r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 15.83r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 16.05r 0.00u 0.00s 0% 3196k
load: 0.03  cmd: ifconfig 23246 [sbwait] 16.28r 0.00u 0.00s 0% 3196k
<full output>

any ideas on how to debug this?

TIA
gtz42
 
Maybe it's caused by NICs passthrough to the bhyve vm. Is it the same if the vm is not running and nic is available on the host?
 
disable all LRO/TSO-offloading on those NICs.
Intel NICs with LRO/TSO enabled completely reset upon *every* connect/disconnect of virtual interfaces (vlan, bridges etc) which can lock up the network for several minutes during startup of VMs/jails.
 
Andriy said:
Start with procstat -kk of the ifconfig process while it's stuck.
Click to expand...

Code: 
  PID    TID COMM                TDNAME              KSTACK
60893 100403 ifconfig            -                   mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f kern_recvit+0x180 sys_recvmsg+0x67 amd64_syscall+0x10e fast_syscall_common+0xf8
 
VladiBG said:
Maybe it's caused by NICs passthrough to the bhyve vm. Is it the same if the vm is not running and nic is available on the host?
Click to expand...
I tried that, but apparently not thoroughly enough.

It's seams to be combination of pci-paththrough and having the device attached in the vm.

It only happens if I just passthrough some interfaces from the i350-AM4 chip.

As soon as the VM attaches any of this interfaces, ifconfig stalls. Even after the VM was stopped, ifconfig still stalls.
 
For the record, this are the relevant lines of pciconf

Code: 
root@box:~ # pciconf -lv
...
igb0@pci0:4:0:0:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:4:0:1:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
ppt0@pci0:4:0:2:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
ppt1@pci0:4:0:3:    class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
...
ix0@pci0:8:0:0:    class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c8 subvendor=0x15d9 subdevice=0x15c8
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553/X557-AT 10GBASE-T'
    class      = network
    subclass   = ethernet
ix1@pci0:8:0:1:    class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c8 subvendor=0x15d9 subdevice=0x15c8
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553/X557-AT 10GBASE-T'
    class      = network
    subclass   = ethernet
ppt2@pci0:10:0:0:    class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15ce subvendor=0x15d9 subdevice=0x15ce
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
ppt3@pci0:10:0:1:    class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15ce subvendor=0x15d9 subdevice=0x15ce
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
...

While I would have liked to keep one of the igb interfaces on the host, the ix1 interface is RJ45 and works with the 1GbE port on the switch.
While a waste, I don't need it at the moment.

I've included all for sub systems of the i350 in pptdevs, and now ifconfig does not stall.
 
One more thing:

ifconfig does not seam to stall, when I split the ix interfaces:
  • vm:10/0/0 + 8/0/1
  • host: 8/0/0 + 10/0/1
But than the vm does not get a link (on 10/0/0, even the switch states the link is down). (10/0/1 and 8/0/1 are currently not wired/connected)

What irritates me is the fact that I've been splitting a (different) i350-AM4 for years into 4 VMs, but using Linux/Proxmox on the host, even occasionally using one of the interfaces on the host (in a LXC).
 
Not directly related to what's being discussed here, just some random facts. I was recently writing a bnxt (Broadcom 57416 and friends) driver for another OS, and all configuration is done on function 0 of that multi-port NIC, so if you would passthrough the function 1 into VM you can't really properly configure it (low-level in driver). I *guess* some of the Intel NICs could have similar design. I would look into using SR-IOV instead.
 
You must log in or register to reply here.
Back
Top