bhyve Can't get bhyve vm to recognize more than 2 cpus on an N150

R

rss2706

I am trying sylve as a vm/jail manager. I am running FreeBSD 15.1-RELEASE on my N150 based Beelink Mini ME with the latest version of sylve (vers 0.2.3.4). My bhyve vm under this configuration is running OPNsense 26.7.1_1 (also FreeBSD 15.1). The settings in sylve are sockets=1, cores=4, threads=1, which matches the physical configuration of my N150 Intel processor. When I boot the OPNsense vm, it only recognizes 2 cores with 2 thread per core. I have performed the following diagnostic steps:

1. Verified the right topology in the host using sysctl
Code: 
kern.smp.topology: 0
kern.smp.cores: 4
kern.smp.threads_per_core: 1
kern.smp.cpus: 4

kern.sched.topology_spec: <groups>
 <group level="1" cache-level="3">
  <cpu count="4" mask="f,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3</cpu>
 </group>
</groups>
2. Verified the proper arguments are being passed by libvirt when creating the vm by examining /var/log/libvirt/bhyve/100.log (100 is the vm number)
Code: 
bhyve -c cpus=4,sockets=1,cores=4,threads=1 -m 2048 -S -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/var/db/sylve/vms/100/100_vars.fd -s 2:0,xhci,tablet -s 1:0,lpc -s 3:0,virtio-net,tap0,mac=ba:97:78:fd:81:ea -s 4:0,virtio-net,tap1,mac=86:0d:5a:76:96:30 -s 5:0,virtio-net,tap2,mac=aa:27:40:11:1c:ce -s 6:0,virtio-net,tap3,mac=fe:53:c0:18:a4:e4 -s 12,virtio-console,org.qemu.guest_agent.0=/var/db/sylve/vms/100/qga.sock -s 13:0,passthru,1/0/0 -s 14:0,fbuf,tcp=127.0.0.1:5957,w=1280,h=960,password=simple-become-thought-coward-reflection-joy -s 10:0,nvme,/dev/zvol/zroot/sylve/virtual-Verified that machines/100/zvol-5 100
3. Verified that the host allows creation of virtual machines with 4 virtual cpus using sysctl
Code: 
hw.vmm.maxcpu: 4
4. This is the result in the OPNsense vm examining sysctl output
Code: 
ern.smp.topology: 0
kern.smp.cores: 2
kern.smp.threads_per_core: 2
kern.smp.cpus: 4

kern.sched.topology_spec: <groups>
 <group level="1" cache-level="3">
  <cpu count="4" mask="f,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3</cpu>
  <children>
   <group level="2" cache-level="2">
    <cpu count="2" mask="3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">2, 3</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
  </children>
 </group>
</groups>
5. I tried disabling hyperthreading in the vm by setting the machdep.hyperthreading_allowed=0. The result was the vm only recognized 2 cores with 1 thread per core (2 vcpus).

When examining the scheduling behavior using top -P in the OPNsense vm with 2 cores, 2 threads per core, the cpu load is weighted to the 2 cpus the vm thinks are physical cores with very little activity occurring on the hyperthread cores. I assume this is due to the schedulers knowledge that the 2 hyperthread cores are not real independent execution units.

Any help in debugging and resolving this issue would be greatly appreciated.
 
Tried on an old i5-3570 and pfSense as bhyve VM, I don't have any problem with cpus=4,sockets=1,cores=4,threads=1, sysctl kern.smp.cores reports well 4.

So, it's either a problem with OPNsense, or more probably, with your processor. You should try a pure FreeBSD VM to sort it out that. If it is the N150 "fault", it may worth a PR.
 
Emrion, thank you for your reply. Per you suggestion, on my other Beelink Mini ME host with the N150 and running a fresh install of FreeBSD 15.1-RELEASE, I installed a vm running FreeBSD 15.1-RELEASE using vm-bhyve. That takes both sylve and OPNsense out of the mix as potential problems. The vm report 2 cores and 2 threads, just like the OPNsense install with sylve. Here is an excerpt from sysctl

Code: 
CPU: Intel(R) N150 (800.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0xb06e0  Family=0x6  Model=0xbe  Stepping=0
  Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
  Features2=0xfeda7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x201407a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,SMAP,SHA>
  Structured Extended Features2=0x400600<VAES,VPCLMULQDQ,RDPID>
  Structured Extended Features3=0x400<MD_CLEAR>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory  = 2147483648 (2048 MB)
avail memory = 2048409600 (1953 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVAPIC >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: L2 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads

In the vm-bhyve jail config file, I am explicitly setting cpu=4, cpu_sockets=1, cpu_cores=4, cpu_threads=1. I also checked the host bios, which recognizes 4 cores with 1 thread per core. Host sysctl clearly shows 4 cores

Code: 
  Origin="GenuineIntel"  Id=0xb06e0  Family=0x6  Model=0xbe  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  Structured Extended Features4=0x810
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x1580fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16401031168 (15641 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L3 data cache covers more APIC IDs than a package (7 > 3)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)

This seems like some kind of issue with how bhyve is getting or interpreting the cpu information.
 
You must log in or register to reply here.
Back
Top