bhyve Windows Server slow IO

I don't see the advantage of passing through an NVMe.
forums.freebsd.org

NVMe passthru on Bhyve findings

I am currently using a SuperMicro X10DRL board for my Bhyve Virtualization machine. OS boots from a 64GB DOM. I use two Samsung 512GB PM953 drives in a M.2 form factor mounted on a SuperMicro AOC-SLG3-2M2. The slot where the card is installed is set to x4x4 in the BIOS for bitfurication. I am...
forums.freebsd.org forums.freebsd.org

From my findings your better off hosting VM's on them and using ahci-hd for the images versus -s 7:0,nvme,/dev/nda2
 
Vm-bhvye is very simplistic. CBSD is robust with support for multiple OS platforms and continues to require more testing from its developers to ensure wide use. Chyves is only for FreeBSD. This background is necessary because I spent the last five days trying to get bhyve run with no issues.

CBSD won't run ISOs from its own repo despite using 'cbsd bconstruct-tui' wizard. Vm-bhvye works with ease but later fails. I got latest Freepbx running on it and "vm console..." showed expected information while "cbsd blogin..." hangs just as it reads its CD, peripherals and boot entries.

The problem with vm-bhvye is that web access is lost after one or two clicks at the initial setup after a fully automatic installation over console.

The vm-public bridge like tap0 keeps changing its ether address at every reboot. What this means is that the [new] MAC address would not be the same as MAC address earlier enter into the VM-BHYVE_DIR/.config/system.conf in host machine AND the etc/sysconfig/network-script/ifcfg-eth0 in the VM itself. When the Ethernet address is the same and the Freepbx web UI is accessible, it is just for few seconds and clicks. Connection via browser will be lost in no time. Usually at this time, the VM could be pinged from outside it e.g. from other jails or host.
Even on creating a bridge connected to tap0 & vm-public and using its static ether address in the above files, the gateway and other IP address cannot be pinged. A passthru of ALL in system.conf did not work either.

Any suggestions would be appreciated.

It is not intended to hijack this thread; it is only coincidental with almost the same problem.
 
Lamia said:
Vm-bhvye is very simplistic. CBSD is robust with support for multiple OS platforms and continues to require more testing from its developers to ensure wide use. Chyves is only for FreeBSD. This background is necessary because I spent the last five days trying to get bhyve run with no issues.

CBSD won't run ISOs from its own repo despite using 'cbsd bconstruct-tui' wizard. Vm-bhvye works with ease but later fails. I got latest Freepbx running on it and "vm console..." showed expected information while "cbsd blogin..." hangs just as it reads its CD, peripherals and boot entries.

The problem with vm-bhvye is that web access is lost after one or two clicks at the initial setup after a fully automatic installation over console.

The vm-public bridge like tap0 keeps changing its ether address at every reboot. What this means is that the [new] MAC address would not be the same as MAC address earlier enter into the VM-BHYVE_DIR/.config/system.conf in host machine AND the etc/sysconfig/network-script/ifcfg-eth0 in the VM itself. When the Ethernet address is the same and the Freepbx web UI is accessible, it is just for few seconds and clicks. Connection via browser will be lost in no time. Usually at this time, the VM could be pinged from outside it e.g. from other jails or host.
Even on creating a bridge connected to tap0 & vm-public and using its static ether address in the above files, the gateway and other IP address cannot be pinged. A passthru of ALL in system.conf did not work either.

Any suggestions would be appreciated.

It is not intended to hijack this thread; it is only coincidental with almost the same problem.
Click to expand...

It's now fixed. Passthru finally worked. Existing documentation is not very clear on where changes are required. I needed to make changes to both the loader.conf and system.conf. Then it goes beyond that to systonically bridge two interfaces in a VM.


In fact, this is just another form of hack that makes me think that FreeBSD is not for the faint-hearted.
 
abishai said:
Does Xeon X56xx support this?
Click to expand...
From what I read it seems that APIC virtualized(APCIv) was first a software feature that was moved onto the CPU.
So maybe E5 Xeon V2 family was the first with APCIv on the CPU.
software.intel.com

Data Center

Access technologies that use data for modern code, machine learning, big data, analytics, networking, storage, servers, cloud, and more.
software.intel.com software.intel.com
 
I'm not sure if this helps anyone, but I had a need to run a Windows server VM again and my host hardware has changed. In the past, I was testing this on an i5 4670 with 8GB of RAM and on an SSD. Most everything worked great but it choked at IO. Unzip a 100kb file? Took 30 seconds. Nasty. I did it again today with the same files and it was instantaneous. Tried again with a 700MB 7z archive and it did it at the speed I expected. Very nice actually, I feel like I can actually recommend bhyve to people now for virtualizing Windows :)

New hardware: Ryzen 3600, 16GB RAM, running my VMs off a zpool that is on a Samsung 840 Pro SSD
OS: 12.1-RELEASE
 
I feel like I can actually recommend bhyve to people now for virtualizing Windows :)
Click to expand...

sure wish pci passthrough worked better for graphics cards .. Id never look back if my 2080ti worked better :( on a windows vm .. perhaps I missed something..

did you end up passing the nvme? or did the new hardware fix the io? or you just created a pool/dataset for vm's on a new drive?
 
zader said:
sure wish pci passthrough worked better for graphics cards .. Id never look back if my 2080ti worked better :( on a windows vm .. perhaps I missed something..

did you end up passing the nvme? or did the new hardware fix the io? or you just created a pool/dataset for vm's on a new drive?
Click to expand...

The OS isn't running on an NVMe drive, I have it on a standard SATA SSD. Maybe the new hardware fixed IO? I guess since this is AMD it could behave differently than an Intel box. I'm using vm-bhyve so I just used what they have

Code: 
loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu="3"
memory="6G"

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
ahci_device_limit="8"

# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="e1000"
network0_switch="public"

disk0_type="ahci-hd"
disk0_name="disk0.img"

# windows expects the host to expose localtime by default, not UTC
utctime="no"
uuid=""
network0_mac=""
 
bendany said:
For windows server, my recommendation config is

1. apply this commit https://reviews.freebsd.org/rS358848
2. using nvme as disk backend or passthough a nvme disk.
3. passthrough a nic instead of virtio-net
4. apply this commit https://reviews.freebsd.org/rS349184
5. apply this commit https://reviews.freebsd.org/rS348779

Windows will as fast as it could be.
Click to expand...

I've Xeon E3 1220 and bhyve work very slow on 12.1.
To test the patches I've upgraded to CURRENT and now bhyve working great!
In order to maintain 12.1 is possible to apply these patches and build only bhyve and vmm?
Tested copying the new files in /usr/src/sys/usr.sbin/bhyve and make, now I have the executable in /usr/obj/. How I can install? Manually move the executable and libraries? And for vmm?

Thanks
 
m1001101 said:
In order to maintain 12.1 is possible to apply these patches and build only bhyve and vmm?
Click to expand...
Please see here: https://forums.freebsd.org/threads/bhyve-windows-server-slow-io.71199/post-456220 -- it only contains two of those patches, but I found the first one is the one making the huge difference for windows guests.

About only building parts of the system, it definitely works for bhyve (just use make in the appropriate source directory), I didn't try to just build the vmm module but built a complete kernel, so maybe this works as well.
 
Zirias said:
Please see here: https://forums.freebsd.org/threads/bhyve-windows-server-slow-io.71199/post-456220 -- it only contains two of those patches, but I found the first one is the one making the huge difference for windows guests.

About only building parts of the system, it definitely works for bhyve (just use make in the appropriate source directory), I didn't try to just build the vmm module but built a complete kernel, so maybe this works as well.
Click to expand...

Work great!

Thanks!
 
Zirias said:
That's because you get them in a braindead format from the linked site, with only one "hunk" consisting of the whole file. They work on 12.1, but I had to apply them manually, using vim ...

Find attached a patch combining #1 and #4 for 12.1 for your convenience ;)
Click to expand...
Pardon my ignorance, but how do I properly apply this patch to the file (I assume from the info it is /usr/src/sys/amd64/vmm/intel/vmx.c that I am patching), and properly rebuild and install only the bhyve components? I see many references to applying patches or diff files, but have come up empty with any instructions on how to apply them.
 
free-and-bsd said:
No, I'm not. At this stage of testing it suits my needs better to use a startup shell script to manually start/stop/destroy a given VM.
Now bhyve says this about nvme type of emulated device:
Code: 
NVMe devices:

                 devpath     Accepted device paths are:
                     /dev/blockdev or /path/to/image or
                     ram=size_in_MiB.

                 maxq     Max number of queues.

                 qsz     Max elements in each queue.

                 ioslots     Max number of concurrent I/O re-
                     quests.

                 sectsz     Sector    size (defaults to blockif sec-
                     tor size).

                 ser     Serial    number with maximum 20 charac-
                     ters.
But I just simply used nvme,/path/to/image, which, obviously, wasn't enough or something. So I wonder if sysutils/vm-bhyve uses some defaults there for maxq, qxz, ioslots, sectsz, ser.
Click to expand...

Yes, bhyve manpage is not so good descript how to use nvme backend.
Maybe I make a patch to manpage later.
Usually, I use these parameters:
Code: 
-s 4:0,nvme,/dev/zvol/zones/vm/nvme1/disk0,maxq=16,qsz=8,ioslots=1,sectsz=512,ser=ABCDEFG

Hope this help.
 
jardows said:
Pardon my ignorance, but how do I properly apply this patch to the file (I assume from the info it is /usr/src/sys/amd64/vmm/intel/vmx.c that I am patching), and properly rebuild and install only the bhyve components? I see many references to applying patches or diff files, but have come up empty with any instructions on how to apply them.
Click to expand...

If patch relate to vmm.c you can rebuild vmm.ko only. if patch relate to bhyve, you can only rebuild bhyve. that make fast.
usally I clone bhyve code from 13-CURRENT. build in 12.1-STABLE. (CURRENT's bhyve has more feature and more bugfix.)
 
I hate to be a debbie downer, but I am too having issues with bhyve on 12.1-RELEASE-p7. It only stays usuable for 10 minutes or so then freezes. I am going to keep my eyes peeled for a patch or fix. Would like to know if anyone else is having this sort of issue? I have tried installing virtio drivers, nvme disk type....no luck.

I am willing to help out as my skill set allows, just message me!
 
joeafterdinner said:
I hate to be a debbie downer, but I am too having issues with bhyve on 12.1-RELEASE-p7. It only stays usuable for 10 minutes or so then freezes. I am going to keep my eyes peeled for a patch or fix. Would like to know if anyone else is having this sort of issue? I have tried installing virtio drivers, nvme disk type....no luck.

I am willing to help out as my skill set allows, just message me!
Click to expand...
Impossible to tell without seeing the command line you use to start your bhyve. The hardware you're using also matters. Any of these things may be the problem.
Other than that, bhyve is being used by many in production environments, so certainly it's not prone to freezing like that.
 
free-and-bsd said:
I have to use VMware Player:
Click to expand...

Are you saying you run VMware Workstation Player with FreeBSD as the host OS?

Via google, the only info I can find is that VMware runs on only Windows Linux and Mac.

If it's faster than VirtualBox with a Windows-10 guest, and has USB-2 passthru, it could be very useful...
 
j77h said:
Are you saying you run VMware Workstation Player with FreeBSD as the host OS?
Via google, the only info I can find is that VMware runs on only Windows Linux and Mac.
If it's faster than VirtualBox with a Windows-10 guest, and has USB-2 passthru, it could be very useful...
Click to expand...
Linux host, not FreeBSD, that's the point. No VMware tools for FreeBSD host and none expected in the future. Which makes it hardly any better than VirtualBox or others.
 
Are the patches described in this post included in 13.0-RELEASE?
I'm in the proces switching our VMs from XCP-NG to FreeBSD bhyve vm.
In case I need to apply the patches, where can I find documentation on how to apply the patches?

Also, what is the current best practice? ahci, virtio-blk or nvme disk type?
The host storage is on 4 disk mirrored vdev.

this is my current config running on HP ProLiant DL360e Gen8 CPU E5-2450L
tried several virtio driver versions, version newer than virtio-win-0.1.187 caused Windows 10 install to fail.

:
loader="uefi" graphics="yes" xhci_mouse="yes" cpu=8 cpu_sockets=1 cpu_cores=8 cpu_threads=1 memory=8G ahci_device_limit="8" network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_opts="sectorsize=512" disk0_name="root" disk0_dev="sparse-zvol" disk1_type="ahci-cd" disk1_dev="custom" disk1_name="/zroot/vm/.iso/sage.iso" utctime="no" uuid="800d78eb-" network0_mac="58:9c"
 
You must log in or register to reply here.
Back
Top