bhyve Windows server with virtio disk in Bhyve does not work

Hi

I have as of FreeBSD 13 (at last :D) moved my personal virtualisation server from CentOS/KVM over to FreeBSD/Bhyve. Works like charm for all VMs, no matter if they use grub, uefi or bhyveloader. Performance, VLAN:s and everything else works perfect as well. FreeBSD 13 performance is *r e a l l y* great. Good job with FreeBSD 13, bhyve etc. Thank you developers!!!

I have however noted a problem that others may have noted too.

I have a windows server 2019 VM that caused a problem during migration over to bhyve. It is OK and usable now, but not optimal. Can live with it, but want to ask the community...

When used KVM on CentOS 8:
Used virtio network driver and virtio disk drivers (for boot disk as well).

I had to install a second disk in W2019 and after that install the virtio disk driver. After that I could shut down the vm, switch the boot disk over to virtio, boot, and it worked permanently.



After going over to Bhyve on FreeBSD 13.0-RELEASE-p2:
The W2019 did not boot, but immediately crashed. I hade to switch from the originally configured disk0_type=“virtio-blk" to disk0_type=“ahci-hd" and go through a recovery process with windows 2019. After that the W2019 worked OK with virtio network driver but with no disk on virtio.

Now… I have tried to add a second separate not boot disk with virtio and kept boot disk as non virtio. I have also tried lastest and a few older virtio disk driver releases. I does not matter what I do, windows 2019 crash during boot. Directly...




I have seen complaint that Microsoft is to blame for this. But it do work flawless in KVM… I do have superfast SSD:s for the OS disk, so it works ok on non virtio disk drivers. But I wonder if there is anybody that have a solution of how to make a virtio disk to work on windows server. Boot disk or not. It does not work at all today. At least not in FreeBSD 13-RELEASE-p2.



Tnx
Peo
 
I have seen complaint that Microsoft is to blame for this.
The Windows installation will disable any and all other disk controller drivers except for the ones it found during installation. So if you change the underlying controller Windows can't do anything else but STOP because it just can't find the disks to boot from. It's the same when you switch from IDE to AHCI in the BIOS. You'll need to enable the AHCI driver before doing that or Windows will just boot into a blue screen.

And I'm honestly not sure if Windows even has drivers for virtio-blk devices. It certainly doesn't have drivers for virtio-net because you need to install those separately.
 
I hear you... But I can for sure see it works in KVM. And the disk performance is better with the virtio. I can live with not using virtio on boot disk as I have super fast SSD:s and a lot of RAM for the VM with ZFS on host that has over 100GB RAM to use :)

But if you are right in this I of course wonder why it works under KVM.


It is simple....

W2019 server on KVM
- Virtio network works
- Virtio boot disk works (but only of you first add a second virtio disk so the driver load.)
- Virtio second disk works

W2019 server on Bhyve FreeBSD 13 release p2

- Virtio network works
- Virtio boot disk does not works
- Virtio second disk does not works
i.e a virtio disk does not work at all here


Note that it doesn't work in KVM either on boot disk unless you add a disk 2 first with virtio. That is the key to use the virtio on boot disk. But do note that in Bhyve it doesnt matter even if you skip virtio on the boot disk and just try to use it on a second not used disk. So I am not entirely sure I agree with you :)

As said. I can live with it. But I of course want to note it so it maybe will work later on in FreeBSD.... And maybe someone here on the forum have done a deeper analysis than me.
 
Addition...

--snip--
And I'm honestly not sure if Windows even has drivers for virtio-blk devices. It certainly doesn't have drivers for virtio-net because you need to install those separately.
--snip--

Yes. As I noted I have installed the needed drivers virtio disk and network drivers separateley. And also tried different versions.
 
Ideally you'd hope it would *just boot* if moving from KVM that was using virtio-blk devices to a bhyve host using the same devices.

Obviously changing drivers is commonly a problem with Windows (god knows why they've never done something about it). Just out of interest have you tried the nvme option? It's the current preferred choice for newer Windows versions as it is usually the highest performing and is supported without third party drivers.
 
Ideally you'd hope it would *just boot* if moving from KVM that was using virtio-blk devices to a bhyve host using the same devices.

Obviously changing drivers is commonly a problem with Windows (god knows why they've never done something about it). Just out of interest have you tried the nvme option? It's the current preferred choice for newer Windows versions as it is usually the highest performing and is supported without third party drivers.

usdmatt You seems to have the same option as SirDice , that it probably is a windows issue even though it works in KVM :) Could be, but the test I have done doesn't go in line with that. But I can have missed something or draw the wrong conclusions based on the facts I have of course. That is why I post :)

Regarding nvme.... I have no nvme disks in the host server. But that maybe does not matter? And how could nvme boost disk performance as virtio? It works stable now when I backed out from virtio-blk to ahci-hd. And I have reasonable performance (well.. really good actually..) with it as the host SSDs are the best and the host disk cache is h u g e :)

This leads to two questions...

Where can I read more about the nvme option for bhyve? Interesting, but I have only seen the text in the bhyve config.example which does not say that much.

Shall I consider windows server virtio disk drivers as dead if using Bhyve ?

Tnx in advance
 
Check the driver version in windows device manager for the storage controller. I experienced trouble with newer drivers, for me the latest working storage driver was in virtio-win-0.1.187
 
Hi:

I just wanted to post here as i had the exact same issue as OP but i managed to resolve it.

Just like OP would bdos when trying to swap to virtio blk drivers for windows 10 enterprise despite following the many guides out there, most of which tell you to create a dummy storage device requiring virtio. And then installing the drivers for them, rebooting and swaping boot partition to virtio.

In the end the issue was simply the virtio driver versions, it seems that the latest 208 stable does not work with bhyve freebsd version 12.2.
When i uninstalled the driver and repeated the above steps with virtio 187, the VM successfully posted.
I wanted to let you all know so most of you dont end up wasting time using latest virtio drivers.

Note: although the vm with 208 did not work with freebsd it did with proxmox. This to me suggest a bhyve issue more than anything else.
 
Back
Top