bhyve bhyve: guest building world causes steady 30% system CPU load on host

H

hpnothp

I have Host FreeBSD 14.3-RELEASE-p2 with single running Guest: FreeBSD 15-CURRENT - where is running make buildworld-jobs It somehow causes constant 30% system CPU usage to host, while inside guest everything looks OK (around 2-10% system CPU).

Question: is there some way to find out what is causing that 30% system CPU load to bhyve host?

My only plausible answer is that CPU used by bhyve guest is reported as 'system' (instead of 'user'). When I divide all (saturated) CPUs for guest (6) by system CPUs (20) and multiply by 100, I get 30%.


Host details:
  • FreeBSD 14.3-RELEASE-p2
  • Intel(R) Core(TM) i5-14500, nproc reports 20
  • 32GB physical memory
  • 2x NVVMe in RAID0 on ZFS
  • running i915 DRM with XFCE Desktop

Host vmstat output while guest is busy building world:

Bash: 
$ vmstat -h 60
 procs    memory    page                      disks       faults       cpu
 r  b  w  avm  fre  flt  re  pi  po   fr   sr nda0 nda1   in   sy   cs us sy id
 6 14 10 592G  11G 1.1k  60   2   3 6.8k 5.9k    0    0 2.0k  12k  19k  0 12 86
 6 14 10 594G  11G 1.1k   0   1   0 1.7k 4.8k   28   18 2.3k 8.2k  14k  0 28 71
 6 14 10 592G  11G  600   0   1   0  481 4.8k   20   13 2.2k 4.6k  10k  0 28 71
 6 14 10 594G  11G  850   0   1   0 1.5k 4.8k   32   17 2.2k  10k  13k  0 23 76
 6 14 10 592G  11G 1.2k   0  57   0  15k 4.8k   19   69 2.4k  16k  30k  0 29 69
 6 14 10 592G  11G  162   0   1   0  25k 4.8k   32   18 2.4k 8.7k  39k  0 29 70
 6 14 10 592G  10G  118   0   0   0  38k 4.8k   24   19 2.4k 5.8k  54k  0 30 69
 7 14 10 592G  11G  109   0   0   0  30k 4.8k   31   23 2.4k 5.6k  43k  0 30 69
 6 14 10 594G  10G  384   0   0   0  37k 4.8k   20   14 2.4k 5.9k  53k  0 30 69
 6 14 10 592G  11G  747   0   2   0  36k 4.7k   30   20 2.4k 8.4k  50k  0 30 69

Notice that system CPU is steady 30% most of time.

Host 2nd output from top:
Bash: 
$ top -b -SHz -s 1 -d 2

(first print skipped)

last pid:  4382;  load averages:  6.26,  5.36,  3.33  up 0+04:48:34    11:44:17
2279 threads:  27 running, 2183 sleeping, 1 zombie, 68 waiting
CPU:  0.4% user,  0.0% nice, 30.7% system,  0.0% interrupt, 68.9% idle
Mem: 11G Active, 1797M Inact, 1362M Laundry, 6794M Wired, 72K Buf, 10G Free
ARC: 4908M Total, 1682M MFU, 3161M MRU, 9797K Anon, 15M Header, 40M Other
     4611M Compressed, 6439M Uncompressed, 1.40:1 Ratio
Swap: 16G Total, 1147M Used, 15G Free, 6% Inuse

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 2981 root        135    0    10G  7432M CPU10   10 123:06 100.00% bhyve{vcpu 4}
 2981 root        135    0    10G  7432M CPU12   12 122:13 100.00% bhyve{vcpu 2}
 2981 root        135    0    10G  7432M CPU18   18 122:07 100.00% bhyve{vcpu 3}
 2981 root        135    0    10G  7432M CPU9     9 122:06 100.00% bhyve{vcpu 0}
 2981 root        135    0    10G  7432M CPU13   13 121:40 100.00% bhyve{vcpu 5}
 2981 root        135    0    10G  7432M CPU4     4 121:23 100.00% bhyve{vcpu 1}
 4159 hp           21    0  2671M   245M select   3   2:33   1.84% firefox{Isolated Web Conten}
 2386 root         21    0   390M   106M select  17   8:49   1.77% Xorg{MainThread}
 2607 hp           20    0    12G  1002M uwait    6   1:14   1.08% firefox{Renderer}
   12 root        -64    -     0B   768K WAIT     8   1:57   0.89% intr{irq149: xhci0}
 2607 hp           20    0    12G  1002M uwait   17   0:20   0.88% firefox{WRRenderBackend#39}
 2469 hp           20    0   262M    52M select   7   1:03   0.56% xfwm4{xfwm4}
   15 root        -60    -     0B    96K -        3   1:23   0.49% usb{usbus0}
 4382 hp           20    0    23M  7944K CPU0     0   0:00   0.36% top
 2616 hp           20    0  3278M   341M select   0   0:56   0.32% firefox{Isolated Web Conten}
 2607 hp           20    0    12G  1002M uwait    0   0:16   0.27% firefox{IPDL Background}
 4159 hp           20    0  2671M   245M kqread   7   0:10   0.26% firefox{IPC I/O Child}
 2607 hp           20    0    12G  1002M uwait    7   0:56   0.25% firefox{glean.dispatcher}

Guest details:
  • using vm-bhyve
  • 6 CPUs, 10GB memory, disk emulation: nvme, network virtio-net

Here is vm-bhyve configuration file:
Bash: 
loader="bhyveload"
cpu=6
memory=10G
network0_type="virtio-net"
network0_switch="default"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="ba07eb0f-773d-11f0-bb63-4cd717a0f39c"
network0_mac="58:9c:fc:08:20:6c"

Guest OS Details:
  • uname -v -> FreeBSD 15.0-PRERELEASE #0 main-n279550-4a94dee2a497: Tue Aug 12 14:03:40 CEST 2025 root@fbsd-next:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  • uname -UK -> 1500059 1500059
  • freebsd-version -> 15.0-PRERELEASE

Guest workload:
Bash: 
$ cd /usr/src
$ make buildworld-jobs

[Creating objdir /usr/obj/usr/src/amd64.amd64...]
make: /usr/src/share/mk/jobs.mk:47: 
@ 1755077623 [2025-08-13 11:33:43] Start buildworld-jobs
@ 1755077623 [2025-08-13 11:33:43] Start buildworld -j1.33  log=/usr/buildworld.log
...

Guest Vmstat - system CPU is varying but mostly under 10% after a while it typically settles to something around 3%:
Bash: 
$ vmstat -h 60
 procs    memory    page                      disks       faults       cpu                                          
 r  b  w  avm  fre  flt  re  pi  po   fr   sr nda0 pas0   in   sy   cs us sy id                                     
 7  0  0 4.0T 3.1G  40k  45  12   1  42k  532    0    0   42 9.3k  805 41  2 55                                     
 7  0  0 4.0T 2.5G  64k   0  39   7  63k  260   49    0   51  12k 1.0k 89  4  6                                     
 5  0  0 4.0T 2.7G  45k   0  11   3  47k  448   29    0   29 5.7k  376 97  2  0                                     
 7  0  0 4.0T 3.2G 179k   0  46   7 202k  108   47    0   48 145k  10k 55 16 28                                     
 7  0  0 4.0T 3.1G 327k   0   3   0 349k  124   30    0   30  54k 2.8k 73 23  2                                     
 6  0  0 4.0T 3.0G 276k   0   0   0 293k  129   21    0   22  46k 2.5k 79 19  0
 7  0  0 4.0T 2.8G 146k   0  43   2 154k  159   36    0   36  32k 2.1k 86 10  2
 7  0  0 4.0T 2.9G 144k   0  32   3 152k  164   32    0   32  33k 2.1k 89 10  0
 7  0  0 4.0T 2.9G 145k   0  10   1 153k  138   19    0   19  31k 1.5k 90  9  0
 7  0  0 4.0T 2.8G 142k   0  68   4 149k  136   55    0   55  38k 1.9k 87 10  2

Please let me know if you need any more details.
 
bhyve(8)
Code: 
     -H          Yield the virtual CPU thread when a HLT instruction is
                 detected.  If this option is not specified, virtual CPUs will
                 use 100% of a host CPU.
 
I'm sorry to not explain clearly what is problem:

My problem is that guest user CPU time is seen as Host system time.

So I would expect on host 30% user CPU time, but it is reported as 30% system CPU time.

It is confusing in regard to monitoring, because system time is generally CPU time spend in kernel - and higher values are usually sign of serious troubles.
 
You must log in or register to reply here.
Back
Top