bhyve A solid HOWTO with BHYVE on FreeBSD 14 and Windows 11

pathiaki

pathiaki

So, now that I've tried Vbox and found it's performance lacking, I've got a lot of feedback that I should try BHYVE. However, I'm finding a lot 'half' instructional documents. I'd like to get BHYVE running on FreeBSD14 HOST and Windows 11 running as a guest.

Some things that I've found is that there doesn't seem to be VNC server that is settled on and a few don't even have startup scripts so you can enable it from /etc/rc.d. Is this correct? Is there a preferred one?
(Everything seems to point to having a VNC server running on default port 5900 so you can start the 'install' of Windows with a VNC viewer by connecting to the VM with the viewer)

I'm a BHYVE newbie if that's not obvious.... be gentle. :D

Paul
 
Problem is, I never used Windows, and my bhyve instances are mainly automated containers for isolation only.
But indeed it seems that the bhyve docs are adressing somebody who already knows weill about the required things for virtualization.

I am not sure how important that VNC thing is, but in my understanding some of the important questions are (from bottom up):
  • how do you wire your networks?
  • how do you configure your virtual hardware (pci-slots, interfaces, etc.)
For this I didn't find configuration tools that were to my liking (commandline is a pain, packages are already too much), so I wrote shell scripts for rc.d, so that all the configuration can be put into rc.conf[.local].
 
I know I'm not addressing the question of "please give me a hand-holding guide" - but what I can say is that I have no problems with Windows 11 under bhyve. I have several Windows 11 bhyve VMs that I use daily and it works pretty well.

I wouldn't recommend using VNC, I usually use RDP. Definitely always with Windows clients but all of my graphical Linux bhyve VMs are using RDP too.

If you're not comfortable managing your VMs manually, I can recommend having a look at sysutils/vm-bhyve.
 
Well, so far, I've followed the Handbook on Bhyve. (I know, crazy, right? ;))

I did the following:

kldload vmm

# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em1 addm tap0
# ifconfig bridge0 up

(exactly from Handbook except my interface on my internal network is em1)

From my /zroot/bhyve directory:

# bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc \
-s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./disk0.img \
-s 4:0,ahci-cd,./Win11_23H2_English_x64v2.iso -c 4 -m 1024M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
windows11

I believe what I'm doing above is calling bhyve with various flags to talk across the bridge on tap0, block interface to disk image from iso on Windows with 4 cores and 1GB of ram and a framebuff that is 1920x1080 that can be connected at tcp port 5900... the UEFI boot interface is required for gfx interfaces OS like Windows.

Paul
 
Thank you both. I'll try the graphical for now but I really want to know the ins and outs of Bhyve. :)

I'm a sysadmin by trade and I like knowing what's under-the-covers. :D

I also wouldn't mind creating a document going forward (aka hand-holding HOWTO). I know we're usually of this type of people but this project should have taken off faster and farther than it has in 30+ years. I believe braindead HOWTOs and GUIs would help that.
 
algi.... definitely a large step forward.

# If you want to pull a graphical console, you'll need the UEFI loader,
# no matter what OS you're installing on the guest.

loader="uefi"
graphics="yes"
xhci_mouse="yes"

# If not specified, cpu=n will give the guest n discrete CPU sockets.
# This is generally OK for Linux or BSD guests, but Windows throws a fit
# due to licensing issues, so we specify CPU topology manually here.

cpu=2
cpu_sockets=1
cpu_cores=2

# Remember, a guest doesn’t need extra RAM for filesystem caching--
# the host handles that for it. 4G is ludicrously low for Windows on hardware,
# but it’s generally more than sufficient for a guest.
memory=4G

# put up to 8 disks on a single ahci controller. This avoids the creation of
# a new “controller” on a new “PCIe slot” for each drive added to the guest.

ahci_device_limit="8"

# e1000 works out-of-the-box, but virtio-net performs better. Virtio support
# is built in on FreeBSD and Linux guests, but Windows guests will need
# to have virtio drivers manually installed.

#network0_type="e1000"

network0_type="virtio-net"
network0_switch="public"

# bhyve/nvme storage is considerably faster than bhyve/virtio-blk
# storage in my testing, on Windows, Linux, and FreeBSD guests alike.

disk0_type="nvme"
disk0_name="disk0.img"

# This gives the guest a virtual "optical" drive. Specifying disk1_dev=”custom”
# allows us to provide a full path to the ISO.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/zroot/bhyve/virtio-win.iso"

# windows expects the host to expose localtime by default, not UTC

utctime="no"

passthru0="4/0/0"
 
jbo@ said:
I know I'm not addressing the question of "please give me a hand-holding guide" - but what I can say is that I have no problems with Windows 11 under bhyve. I have several Windows 11 bhyve VMs that I use daily and it works pretty well.

I wouldn't recommend using VNC, I usually use RDP. Definitely always with Windows clients but all of my graphical Linux bhyve VMs are using RDP too.

If you're not comfortable managing your VMs manually, I can recommend having a look at sysutils/vm-bhyve.
Click to expand...
Have you installed them from scratch or were they P2V (physical to virtual) migrations? I'm considering a P2V migration.
 
I have official ISOs (plural) as well but the pain of reinstalling everything is putting me off. It's currently a separate partition on my laptop. I want to move it to a zvol and later reclaim the space for an additional FreeBSD image.

I did the P2V thing with XP from a physical partition to VirtualBox back in the day. That required a binary edit of the XP boot record and two XP O/S files. The edits removed the disk geometry from the boot record and the two files so XP didn't expect certain data to be at specific addresses on disk. I don't know if Windows 10 or 11 will work. My first attempt at Windows 10 P2V failed to boot.
 
HI all!!!

So, going back to my template file that I pasted above.... I did the following before running it from scottro's page:

vm_enable="YES"
vm_dir="zfs:zroot/bhyve

I copied my template to .templates/windows11.conf

vm create -t windows11 -s 256G windows11

I ran 'vm list' and it shows me:

NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
windows11 default uefi 2 4G - No Stopped

I thought that seemed ok. I then decided (still in zroot/bhyve) to install my ISO.

vm install windows11 ./Win11_23H2_English_x64v2.iso

It came back with:

Starting windows11
* found gues in /zroot/bhyve/windows11
! guest appears to be running already

I ran 'vm list' and it shows me:

NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
windows11 default uefi 2 4G - No Stopped

Any ideas all? (If I need to 're-initialize', and try again, how do I?) How does the template file look?
 
I had some trouble running my W10 but now it works fine. I tried first with vm-bhyve but I found out that it was MUCH easier (to me, personal preference) using bhyve directly, so I simply use a script.

I followed the handbook, some threads here and tada ?.
The logic is very KISS :
Enable bhyve, ifconfig to bridge your physical LAN/WAN to a virtual interface, truncate a hard drive file if it does not exist (zfs or in any local or remote location), run a single bhyve command according to the host needs (for Windows, uefi fd firmware file, 4GB or more of RAM, lpc to 31, keyboard layout if you use a non US keyboard, and virtio drivers for drive/network controller, don't play with sockets because Microsoft doesn't like running multi-socket if you didn't pay for), connect to it using vncviewer then destroy the VM when you turned it off.

PS : Don't forget your firewall if you use one, pf was blocking vnc in my case.
 
Did you, in your config file, have
bhyve_options="-A"
(I usually put at the end, of the file, don't know if that matters).
I believe you are going to need that line, and I don't see it in your post above, unless I overlooked it.
 
scottro said:
Did you, in your config file, have
bhyve_options="-A"
(I usually put at the end, of the file, don't know if that matters).
I believe you are going to need that line, and I don't see it in your post above, unless I overlooked it.
Click to expand...
For me yes, but I pass it as a command argument like the handbook does :
bhyve -AHP (rest of the command).
I'm not on my computer now so I can't send you the full command...

Edit : here it is (I use variables; I'm not expert so it can be greatly enhanced ; run as root)
Code: 
bhyve -AHP \
-s 0:0,hostbridge \
-s 1:0,virtio-net,tap0 \
-s 2:0,ahci-cd,$CD1 \
-s 3:0,ahci-cd,$CD2 \
-s 4:0,virtio-blk,$HD \
-c 4 -m 4096M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,$UEFI \
$VM
My variables point all in one directory on my NAS, containing every required file (the ISOs, the disk IMG, and the UEFI firmware FD file), no ZFS used.
I use 2 CD drives to run both Windows ISO for install and VirtIO ISO for drivers (required to install the SATA drive, as Windows setup will not find any fixed drive + network driver once Windows is setup).
This setup only requires on FreeBSD for vmm to be enabled (vm_path is useless) and the network bridge, if required, and vncviewer. vm-bhyve is NOT used. As I said, some people prefer this way, some prefer vm-bhyve.
On my side, I'll have to add -K to add a French keyboard layout. If you're not using US keyboard, you'll have to do the same because you'll get QWERTY keyboard even if region settings are correct on your FreeBSD and in Windows.

To access the vm, simply run as root or regular user
Code: 
vncviewer 0:5900
(watch for firewalls !)
To check if something is running, you can test
Code: 
sysctl kern.vm_guest
and
Code: 
ls -al /dev/vmm
Once you stopped the VM, run
Code: 
bhyvectl --destroy --vm=$VM
. To restart the VM, just run once again the bhyve -AHP etc...
 
Hello, My system is FreeBSD 14.0-RELEASE-p6, I got
Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz (2000.29-MHz K8-class CPU)
and VT-x: HLT,PAUSE VT-x, enabled by modified the bios.

It was enabled before upgrading the bios to latest. The lasted bios does not have the option and vt is disable by default. Somehow I managed to flash a mod bios to re-enable the VT-x. I also use symcmos tool to modify the register of bios setting from 000 to 001 which enabled the VT.

My dmesg shows VT-x: HLT,PAUSE
I can run virtualbox with no problem.

However when I try to run vm-bhyve,
sh: 
service vm start
. I got
sh: 
/usr/local/sbin/vm: ERROR: kernel vmm not initialised (no VT-x / AMD SVM cpu support?)

I put vm_disable_host_checks="YES" in rc.conf
vm starts however whenever I run vm command it give me error like this
sh: 
/usr/local/sbin/vm: ERROR: unable to load vmm.ko!
I tired sudo kldunload vmm.
I manged to this step
sh: 
sudo vm install void void-live-x86_64-20240314-base.iso
Starting void
  * found guest in /vm/void
  * booting...
sh: 
sudo vm list
NAME  DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
void  default    uefi    2    1G      -    No    Stopped
vm is not running. I guest there is something wrong with the VT-x I could not run vm-BHYVE. I follow this https://vermaden.wordpress.com/2023/08/18/freebsd-bhyve-virtualization/
and this https://srobb.net/vm-bhyve.html and my vm is not running I could not connect to it using vnc viewer

Any ideas? Thanks
 
algi said:
I use vm-bhyve to install Windows according to their wiki - https://github.com/churchers/vm-bhyve/wiki/Running-Windows

Also, I'm using VNC viewer without any issues. Any specific problems you're facing?
Click to expand...

As an issue that I keep running into... I'm having a problem connecting via vncviewer. However, I think it's due to the fact that the machine is waiting on input from the boot.

I have this for 'vm list'

NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
winguest default uefi 2 4G - No Stopped

As you can see, it's 'stopped' and there's no VNC port available to connect. :(
 
pathiaki said:
Well, so far, I've followed the Handbook on Bhyve. (I know, crazy, right? ;))

I did the following:

kldload vmm

# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em1 addm tap0
# ifconfig bridge0 up

(exactly from Handbook except my interface on my internal network is em1)

From my /zroot/bhyve directory:

# bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc \
-s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./disk0.img \
-s 4:0,ahci-cd,./Win11_23H2_English_x64v2.iso -c 4 -m 1024M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
windows11

I believe what I'm doing above is calling bhyve with various flags to talk across the bridge on tap0, block interface to disk image from iso on Windows with 4 cores and 1GB of ram and a framebuff that is 1920x1080 that can be connected at tcp port 5900... the UEFI boot interface is required for gfx interfaces OS like Windows.

Paul
Click to expand...

I also followed the handbook, and everything is working fine. However, there's one thing I'm not sure about: I don't know how to make these changes persistent after a reboot. I think "kldload vmm" should be added to /boot/loader.conf, or maybe in rc.conf? Also, I'm not sure how to set up the bridge to start automatically at boot.

sh: 
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em1 addm tap0
# ifconfig bridge0 up
 
jbo@ said:
I know I'm not addressing the question of "please give me a hand-holding guide" - but what I can say is that I have no problems with Windows 11 under bhyve. I have several Windows 11 bhyve VMs that I use daily and it works pretty well.

I wouldn't recommend using VNC, I usually use RDP. Definitely always with Windows clients but all of my graphical Linux bhyve VMs are using RDP too.

If you're not comfortable managing your VMs manually, I can recommend having a look at sysutils/vm-bhyve.
Click to expand...
Hello, jbo@,

Just out of curiosity, what options are you running for your Win11 VM's? Granted, I'm using vm-bhyve, but I've just experienced so much lag (and I do mean LAG!) when I try to use Win11 VM's with Bhyve. I originally started with 4 vCPU's and 8GB RAM (virtio-net and nvme options) and it was unusable. Then doubled the resources to 8 vCPU's and 16GB RAM and it did help a bit, but still fairly worthless. I even tried doubling that to 16 vCPU's and 32GB RAM and, again, a little better but still what I wouldn't consider "usable." I'm guessing I'm missing something here, just not sure what it is.

Also, I am booting with graphics enabled (only to have access in case something fails,) but I'm connecting via RDP. My Win10, FBSD, OBSD, Linux, etc... VM's all work perfectly fine over RDP. This only appears to be an issue with Win11. I even disabled all the window special effects (which also helped a little.) But, still just so laggy and slow. I see others having this problem as well, but nothing I've seen as far as a solution has cropped up. Any information you could provide may be very helpful to many of us out here struggling with laggy Win11 VM's in Bhyve!

Regards,
Janky Jay, III
 
ek said:
Just out of curiosity, what options are you running for your Win11 VM's?
Click to expand...
Sure, see below.

Just to be clear: I am running several Windows 11 guests with graphics enabled and using RDP to actually interact with the client. I am doing this on different hosts with different hardware ranging from an i7-1185 laptop all the way to a Threadripper 7960X.
Guest CPU and RAM settings range from 4 cores and 8 GB of RAM all the way to 32 cores and 64 GB of RAM.
In some scenarios I use sysutils/vm-bhyve while in other scenarios I "manually" handle bhyve.

Here's one of my vm-bhyve configs (UUID & MAC omitted):
Code: 
loader="uefi"
graphics="yes"
graphics_res="1920x1080"
xhci_mouse="yes"
debug="no"

# CPU
cpu=16
cpu_sockets=1
cpu_cores=8
cpu_threads=2

# Memory
memory=32G

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
ahci_device_limit="8"

# Networking
network0_type="virtio-net"
network0_switch="public"

# disk
disk0_type="nvme"
disk0_name="disk0.img"

# Windows expects the host to expose localtime by default, not UTC
utctime="no"

# Passtrhu
#passthru0="71/0/0"
While the VM specs are "beefy", I have a similar Windows11 vm-bhyve guest with just 4 CPUs and 8 GB of RAM without any issues.

Here's one of my "manual" setups:
Code: 
bhyve \
    -c sockets=1,cores=3,threads=2 \
    -s 0,hostbridge \
    -s 3:0,virtio-blk,/dev/zvol/path/to/windows11_dev/disk \
    -s 10,virtio-net,tap0 \
    -s 31,lpc \
    -s 29,fbuf,tcp=0.0.0.0:5900,w=1280,h=1024 \
    -s 30,xhci,tablet \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
    -m 8G \
    -H -w \
    windows11_dev

Some remarks:
  • Consider disabling any form of power scaling (such as powerdxx) (at least as a debug/test).
  • If you're using ZFS on both the host as well as inside the guest, you want to set the datasets primarycache property to metadata to prevent competing memory pressure.
  • While I have only done this with Windows Server guests, I think there's some merrit in setting the CPU set / affinity to prevent two virtual CPU cores to end up as threads on the same physical CPU core. This can potentially be bypassed by just not letting the guest do hyperthreading.
  • Generally, your Windows guest doesn't need as much memory as a standalone/baremetal Windows installation would as it doesn't need nearly as much RAM for filesystem caching/buffering (this is done by the host filesystem). I have seen Windows 10 VMs running fine with just 2 GB of guest memory. 4 GB should be plenty for regular "office activities".
 
jbo@ said:
Sure, see below.

Just to be clear: I am running several Windows 11 guests with graphics enabled and using RDP to actually interact with the client. I am doing this on different hosts with different hardware ranging from an i7-1185 laptop all the way to a Threadripper 7960X.
Guest CPU and RAM settings range from 4 cores and 8 GB of RAM all the way to 32 cores and 64 GB of RAM.
In some scenarios I use sysutils/vm-bhyve while in other scenarios I "manually" handle bhyve.
Click to expand...
Thank you! Granted, I'm only running this one Win11 VM on an older Dell server (2.2GHz E5-2660's) so I don't expect crazy performance. But, at least usable would be nice.

jbo@ said:
Here's one of my vm-bhyve configs (UUID & MAC omitted):
Click to expand...
Hrm. This doesn't look too different than what I've tried (see below for an example.) I've tried with both single and multi-core configs, threaded and non-threaded. It all seems to perform the same. No noticeable difference. Still laggy.
Code: 
loader="uefi"
graphics="yes"
xhci_mouse="yes"
graphics_res="1600x900"
graphics_wait="no"

cpu=8
#cpu_sockets=2
#cpu_cores=4
#cpu_threads=1
memory=16G

ahci_device_limit="8"

network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0.img"

utctime="no"

jbo@ said:
Some remarks:
  • Consider disabling any form of power scaling (such as powerdxx) (at least as a debug/test).
  • If you're using ZFS on both the host as well as inside the guest, you want to set the datasets primarycache property to metadata to prevent competing memory pressure.
  • While I have only done this with Windows Server guests, I think there's some merrit in setting the CPU set / affinity to prevent two virtual CPU cores to end up as threads on the same physical CPU core. This can potentially be bypassed by just not letting the guest do hyperthreading.
  • Generally, your Windows guest doesn't need as much memory as a standalone/baremetal Windows installation would as it doesn't need nearly as much RAM for filesystem caching/buffering (this is done by the host filesystem). I have seen Windows 10 VMs running fine with just 2 GB of guest memory. 4 GB should be plenty for regular "office activities".
Click to expand...
Again, thank you for the info.
  • I disabled all the power scaling I could find in the VM (at the registry level.) Nothing seems different. Maybe a little speed-up?
  • I'm currently only using ZFS on the host. The VM's I've been testing are all raw files at the moment. Great advice for me to keep in mind, though!
  • I haven't checked the host CPU affinity settings, but I generally do set guests to not using hyperthreading. More good advice!
  • Yep. I have no issues with any other OS VM that I've tested. Most of them run with 2 vCPU's and 4GB RAM. Some 4x8 or 4x12 depending on what they're for, but even under high load, they perform just fine. It's just this dang Win11 VM I can't seem to grasp. I considered maybe it was network related, but why would all the other guests be fine on the same host with the same bridge? I dunno. I'll keep plugging away.
Again, thank you for the info and recommendations. They're greatly appreciated!

- Janky Jay, III
 
I figured it out! Well, sort of. It appears you can't just use cpu=8 as anything above cpu=2 slows things down to a crawl. I found this out by simply setting cpu=2 and restarting and it was about 300% faster. So, I started tweaking and playing with socket, cores, and threads counts to see what I could get away with.

Right now:
Code: 
cpu=8
cpu_sockets=1
cpu_cores=4
cpu_threads=2
memory=16G
... appears to be working just fine. It's still not as responsive as other VM's over RDP, but it's much better than it was and at least usable. Thanks again for your input, jbo@!

- Janky Jay, III
 
I am glad to hear that you got it working!

I forgot to point that out explicitly but yeah: With Microsoft Windows you have to specify the sockets, cores and threads explicitly.
As per my recollection, if you only specify the number of CPUs, bhyve creates just N sockets with one core each.
This creates all sorts of issues with Windows guests. The regular Windows desktop editions simply do not support multiple sockets (or more than 2?) and with the Server editions require you to license per socket. Therefore, you end up in a situation where your Windows guest just has one (or two?) CPUs available.
Or something along those lines... not a Windows expert - just remembering this from past troubleshooting sessions.
 
You must log in or register to reply here.
Back
Top