Hello,
I'm trying to use FreeBSD with Xen, both as a dom0 and domU. To get the inevitable question out of the way, I am not using jails because I wish to run Linux and Windows in the future, and not using bhyve, as Xen seemed to have a better feature set. The setup has gone well, and I have four mostly happy FreeBSD domU's set up, ready to be put into action. The only problem is all disk I/O in the domU's is bottlenecked somewhere to around 20MB/s
Currently, all of my disks are configured as ZFS volumes on the dom0. I have tried multiple ways of attaching these volumes to the guests. I originally tried using them as raw files inserting "raw" into the list, between the path and guest device type. I then switched to using the phy: tag, as it's supposedly designed for block devices over files. For guest device type, I've tried the xvd* paravirt and sd* SCSI controller systems. Neither of them performed above 20MB/s, and the sd* device was even slower.
The server is an HP Proliant DL580 G7 with 4x10 core Xeon E7-4870's and 128GB of RAM. The storage is comprised of two sets of four SAS drives attached to a SAS 6gbps HBA on an add-in card, and arranged into two RAIDZ1 pools. I also have a completely separate NVMe SSD mounted on an M.2 card. The domU's all run FreeBSD 13.0, while the dom0 runs FreeBSD 12.2, due to a kernel panic when loading the sfxge driver (for my 10GbE NIC) in 13.0. The VM's I've been using have had 8 vCPU's and 16GB of RAM assigned.
I determined a rough throughput by doing "pv bigfile > newbigfile" and watching the speed. To ensure this worked, I did this on my desktop Linux machine on an even faster NVMe SSD, and got sustained 1.5GB/s on ext4.
To make sure it wasn't a hardware issue, I booted into FreeBSD without Xen, and tested speed across devices. The slowest speed I ever got, writing to one of my disk pools, was 250MB/s, but I could easily get bursts of up to 1.5GB/s on the SSD, and sustained speeds of 700-1000MB/s. Regardless, this is faster than 20MB/s. I also repeated the test in the dom0, and while it was a bit slower, I still got similar results.
When I installed my VMs, I randomly chose ZFS, in case I wanted to use any of its features, but have since replaced it all with UFS when problems arose. This made no real difference.
I may be forgetting a thing I tried, but this is pretty much it. Xen really is ideal for my use case, and if it weren't for the disk problems, it's already pretty much perfect the way it is. If anyone can give me any advice as to what might be going wrong here, I would massively appreciate it, and I thank all for even taking the time to read this.
I'm trying to use FreeBSD with Xen, both as a dom0 and domU. To get the inevitable question out of the way, I am not using jails because I wish to run Linux and Windows in the future, and not using bhyve, as Xen seemed to have a better feature set. The setup has gone well, and I have four mostly happy FreeBSD domU's set up, ready to be put into action. The only problem is all disk I/O in the domU's is bottlenecked somewhere to around 20MB/s
Currently, all of my disks are configured as ZFS volumes on the dom0. I have tried multiple ways of attaching these volumes to the guests. I originally tried using them as raw files inserting "raw" into the list, between the path and guest device type. I then switched to using the phy: tag, as it's supposedly designed for block devices over files. For guest device type, I've tried the xvd* paravirt and sd* SCSI controller systems. Neither of them performed above 20MB/s, and the sd* device was even slower.
The server is an HP Proliant DL580 G7 with 4x10 core Xeon E7-4870's and 128GB of RAM. The storage is comprised of two sets of four SAS drives attached to a SAS 6gbps HBA on an add-in card, and arranged into two RAIDZ1 pools. I also have a completely separate NVMe SSD mounted on an M.2 card. The domU's all run FreeBSD 13.0, while the dom0 runs FreeBSD 12.2, due to a kernel panic when loading the sfxge driver (for my 10GbE NIC) in 13.0. The VM's I've been using have had 8 vCPU's and 16GB of RAM assigned.
I determined a rough throughput by doing "pv bigfile > newbigfile" and watching the speed. To ensure this worked, I did this on my desktop Linux machine on an even faster NVMe SSD, and got sustained 1.5GB/s on ext4.
To make sure it wasn't a hardware issue, I booted into FreeBSD without Xen, and tested speed across devices. The slowest speed I ever got, writing to one of my disk pools, was 250MB/s, but I could easily get bursts of up to 1.5GB/s on the SSD, and sustained speeds of 700-1000MB/s. Regardless, this is faster than 20MB/s. I also repeated the test in the dom0, and while it was a bit slower, I still got similar results.
When I installed my VMs, I randomly chose ZFS, in case I wanted to use any of its features, but have since replaced it all with UFS when problems arose. This made no real difference.
I may be forgetting a thing I tried, but this is pretty much it. Xen really is ideal for my use case, and if it weren't for the disk problems, it's already pretty much perfect the way it is. If anyone can give me any advice as to what might be going wrong here, I would massively appreciate it, and I thank all for even taking the time to read this.
