bhyve Running Windows in bhyve in a jail

N

Notamoosen

So I've recently gotten back into FreeBSD after not having used it for several years. As a project, to help familiarize myself again, I'm attempting this scenario. I couldn't find any specific instructions so I figured it would be a good challenge. I'll come here periodically to update my journey with the steps that worked, to read advice, and mostly to discuss the "gotchas" I've come across.

2026-05-14
To start I've rented a 6 core, 64GB, 1TB AMD machine from Hetzner. First challenge was getting FreeBSD 15 installed as they no longer officially support it as an image. This was pretty straightforward. In their management panel they have an option to request temporary kvm access. Their kvm allow you to mount an iso and complete the install. One step I did differently from the past was to use ZFS instead of UFS for the first time.

From there I completed my basic setup zsh (with ohmyzsh), doas access, and disabling root over ssh.

Then came setting up the first jail. I'm good on storage space, so I elected to go with a vnet thick jail so I don't have to worry about it conflicting with any future jails. It's also goijg to use a private ip as my isp currently only support ipv4. The handbook was excellent here and worked just as expected. My first gotcha popped up. I learned when the Ethernet interface has external ip do not add it to the bridge. Also, even though I'm using Hetzner's firewall, I still had to install pf to handle nat for the jails. I may add a Google drive link with samples to the files mentioned in this.

I'll continue updating as I pursue further.
 
Notamoosen said:
and disabling root over ssh.
Click to expand...
It's disabled by default.

Notamoosen said:
I learned when the Ethernet interface has external ip do not add it to the bridge.
Click to expand...
Well, you don't want to assign the same IP address to multiple interfaces at all as that would create an IP conflict. On 15 you should put the 'external' IP address on the bridge, not the uplink ethernet interface.
 
Progress!

Once I had the jail working correctly I went to work on bhyve. I started with the handbook for installing bhyve in a jail and it helped me do a bulk of the config. Here's the various gotcha's I came across. Some of these may not be necessary to make it work, but it's part of my currently working setup.

Some host config
add vmm and nmdm to the kernel and make persistent in /boot/loader.conf

I made sure to add the following lines to the jail.conf
allow.vmm;
allow.mount;
allow.mount.zfs;
enforce_statfs=1;

Also I created the zfs dataset and volume I'm planning on using from the host so I added the following to jail.conf to make sure they mount
exec.poststart += "zfs jail bhyve zroot/datase && jexec $name zfs mount -a";
Then I went to the host host and ran the following on the needed dataset.
zfs set jailed=on zroot/dataset

Lastly here is where I was stuck for a while. When trying to start a vm I kept getting directory not found. The handbook mentions creating devfs.rules and mapping them via the jail.conf. It took some digging but I also had to add these lines for it to work correctly.
add path 'zfs' unhide
add path 'vmmctl' unhide

A quick reset of the devfs service and the jail and I'm able to start bhyve in the jail. I've also decided to use the built-in jail and bhyve commands as I find it easier to troubleshoot. I can load Linux iso's successfully but am getting an error when trying to install Windows 11. I'll continue to post more once I work out the issues there.
 
Nice progress. I've collected some links over the years about running Windows under bhyve. Full disclosure: I have not tested any of these or done it myself. I haven't yet had the need to run Windows in a VM. However, I presume everyone involved in the authoring of these notes has had a successful result. Maybe these will help you.
  1. Forum Post
  2. Churchers Github Page
  3. Srobb Blog Post
The forum post mentions (2) and (3).

If you find these of value or you find problems, please let me know so I can update my stashed links.
 
Success!

I certainly came across my share of gotcha's, but it is up and working! I decided to load up Windows 10 instead of 11 so I didn't have to worry about side stepping the hardware requirements. After all, this project was to help me learn about FreeBSD not Windows.


So first off, I couldn't get Windows to boot if I used more than one vcpu. I was able to solve this by adding -w to the bhyve commands. From there I was able to start the machine. It's ultimately going to be acting as a game server so I assigned 4vcpu and 24GB of ram without issue.


For network connectivity I created a tap interface and a bridge inside the jail. I assigned the bridge another ip from the available subnet established earlier and added the tap and epair interfaces to it. Inside windows I configured it's interface with yet another static ip and the network came up.


Next step was automating the startup. I essentially took the bhyve command and arguments I was using to manually start and created a vmrun.sh file from. I made sure to delete the vnc line and add an & to the end of the script to get it to run in the background. From there I created a service under rc.d that could call this file. I found some sample service files online to use. Then it was a matter of enabling the service.


Only thing that was left was to automate the shut down process. This certainly wasn't as smooth as seen in other virtualization solutions, but wasn't too hard to work around. I also suspect if I had used vm-bhyve I may have avoided this as well.


I had to add a line to the jail.conf to "destroy" the vm when the jail shuts down. This prevented an error at startup where it would tell me it existed. I also created an rc.shutdown.local file inside the jail to send a graceful shutdown message to the vm.


Below is a link with the main config files I used for reference.


And there it is, a jailed bhyve windows instance! Some takeaways; while it takes more effort to set up than other solutions, I believe the effort is worth it for the modularity. In my professional life I typically work with Windows and Linux (Almalinux). Everyone is familiar with the issues in Windows of seemingly unrelated activities interfering with each other. Even in Linux I've run into issues where systemd can prevent multiple services from running. I'm not expecting these issues here thanks to how modular everything is.


Also while containers are a great way to get services up and running quickly, I greatly prefer jails. One frustration I come into with containers is that when using a pre-built container, the underlying os may be different than the host. It makes customizing a bit more difficult. Sure, I can start with a base Almalinux to build my own container but the process isn't nearly as smooth as with jails. I like the ability to interact directly with the jail files from the host.


Please don't take this as a knock on other OS's either. I believe they all excel in different ways, and Linux and Windows'ecosystems make them excellent choices, particularly in the business world. However, when it comes to control and stability FreeBSD excels.


So that's that. I hope the information here and in the replies prove helpful to someone. Thanks everyone for the advice given as I certainly used bits and pieces from almost all of it.


Now it's onto my next project, moving my web and email services to this machine. Looking at email/groupware first with possibly iRedMail as the solution, but we'll see.


FreeBSDBhyveJail - Google Drive

drive.google.com drive.google.com
 
You must log in or register to reply here.
Back
Top