I am currently using a SuperMicro X10DRL board for my Bhyve Virtualization machine. OS boots from a 64GB DOM.
I use two Samsung 512GB PM953 drives in a M.2 form factor mounted on a SuperMicro AOC-SLG3-2M2.
The slot where the card is installed is set to x4x4 in the BIOS for bitfurication.
I am using a
This is an older generation of NVMe with not so fast performance. Slightly more than double my SSD speeds.
So I read that FreeBSD 12 includes the option to passthru a NVMe drive with the standard convention:
-s 7:0,nvme,/dev/nda2
For VM passthru testing I am using an additional NVME, a Samsung PM983 in the U.2 form factor, while still hosting my VM's on the gmirror.
Below is testing the newer Samsung PM983 NVMe on the host machine (hypervisor) for a reference point.
So as you can see the speed is almost double, compared to the earlier PM953, on the host before passthru.
Unfortunately, after passthru, the speed drops off considerably. This test is from the VM.
That is even slower than my NVMe based VM image files which are translated to ada0 from the host gmirror NVMe:
As you can see the drop here is around half the speed while a directly passthru NVMe drops the speed by more than 6x.
I am open to suggestions for better speed.
Originally I had planned to convert all my VM's to NVMe passthru. No sense if it is slower.
I use two Samsung 512GB PM953 drives in a M.2 form factor mounted on a SuperMicro AOC-SLG3-2M2.
The slot where the card is installed is set to x4x4 in the BIOS for bitfurication.
I am using a
gmirror
of the two NVMe for redundancy. I mount the gmirror 'tank' at /vm via fstab and have all my VM image files on the mirror.This is an older generation of NVMe with not so fast performance. Slightly more than double my SSD speeds.
Code:
root@virt:~ # diskinfo -t /dev/mirror/tankp1
/dev/mirror/tankp1
512 # sectorsize
960197083136 # mediasize in bytes (894G)
1875384928 # mediasize in sectors
512 # stripesize
0 # stripeoffset
116737 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
Yes # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.032662 sec = 0.131 msec
Half stroke: 250 iter in 0.031611 sec = 0.126 msec
Quarter stroke: 500 iter in 0.062575 sec = 0.125 msec
Short forward: 400 iter in 0.048904 sec = 0.122 msec
Short backward: 400 iter in 0.048604 sec = 0.122 msec
Seq outer: 2048 iter in 0.088228 sec = 0.043 msec
Seq inner: 2048 iter in 0.095934 sec = 0.047 msec
Transfer rates:
outside: 102400 kbytes in 0.097920 sec = 1045752 kbytes/sec
middle: 102400 kbytes in 0.098955 sec = 1034814 kbytes/sec
inside: 102400 kbytes in 0.098688 sec = 1037613 kbytes/sec
So I read that FreeBSD 12 includes the option to passthru a NVMe drive with the standard convention:
-s 7:0,nvme,/dev/nda2
For VM passthru testing I am using an additional NVME, a Samsung PM983 in the U.2 form factor, while still hosting my VM's on the gmirror.
Below is testing the newer Samsung PM983 NVMe on the host machine (hypervisor) for a reference point.
Code:
# diskinfo -t /dev/nda2
/dev/nda2
512 # sectorsize
960197124096 # mediasize in bytes (894G)
1875385008 # mediasize in sectors
512 # stripesize
0 # stripeoffset
SAMSUNG MZQLB960HAJR-000AZ # Disk descr.
S3VKNE0KA06254 # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.017148 sec = 0.069 msec
Half stroke: 250 iter in 0.016615 sec = 0.066 msec
Quarter stroke: 500 iter in 0.031590 sec = 0.063 msec
Short forward: 400 iter in 0.022771 sec = 0.057 msec
Short backward: 400 iter in 0.022195 sec = 0.055 msec
Seq outer: 2048 iter in 0.047863 sec = 0.023 msec
Seq inner: 2048 iter in 0.041782 sec = 0.020 msec
Transfer rates:
outside: 102400 kbytes in 0.056853 sec = 1801136 kbytes/sec
middle: 102400 kbytes in 0.052590 sec = 1947138 kbytes/sec
inside: 102400 kbytes in 0.065681 sec = 1559051 kbytes/sec
So as you can see the speed is almost double, compared to the earlier PM953, on the host before passthru.
Unfortunately, after passthru, the speed drops off considerably. This test is from the VM.
Code:
root@freebsd1:~ # diskinfo -t /dev/nvd0
/dev/nvd0
512 # sectorsize
960197124096 # mediasize in bytes (894G)
1875385008 # mediasize in sectors
0 # stripesize
0 # stripeoffset
bhyve-NVMe # Disk descr.
NVME-5-0 # Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.060528 sec = 0.242 msec
Half stroke: 250 iter in 0.058457 sec = 0.234 msec
Quarter stroke: 500 iter in 0.116592 sec = 0.233 msec
Short forward: 400 iter in 0.090369 sec = 0.226 msec
Short backward: 400 iter in 0.089503 sec = 0.224 msec
Seq outer: 2048 iter in 0.385192 sec = 0.188 msec
Seq inner: 2048 iter in 0.382445 sec = 0.187 msec
Transfer rates:
outside: 102400 kbytes in 0.343736 sec = 297903 kbytes/sec
middle: 102400 kbytes in 0.344296 sec = 297419 kbytes/sec
inside: 102400 kbytes in 0.343548 sec = 298066 kbytes/sec
That is even slower than my NVMe based VM image files which are translated to ada0 from the host gmirror NVMe:
Code:
root@freebsd1:~ # diskinfo -t /dev/ada0
/dev/ada0
512 # sectorsize
80530636800 # mediasize in bytes (75G)
157286400 # mediasize in sectors
32768 # stripesize
0 # stripeoffset
38550 # Cylinders according to firmware.
16 # Heads according to firmware.
255 # Sectors according to firmware.
BHYVE SATA DISK # Disk descr.
BHYVE-9134-7524-5C08 # Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Not_Zoned # Zone Mode
Seek times:
Full stroke: 250 iter in 0.109487 sec = 0.438 msec
Half stroke: 250 iter in 0.105601 sec = 0.422 msec
Seq outer: 2048 iter in 0.376304 sec = 0.184 msec
Seq inner: 2048 iter in 0.401486 sec = 0.196 msec
Transfer rates:
outside: 102400 kbytes in 0.222996 sec = 459201 kbytes/sec
middle: 102400 kbytes in 0.253095 sec = 404591 kbytes/sec
inside: 102400 kbytes in 0.233424 sec = 438687 kbytes/sec
I am open to suggestions for better speed.
Originally I had planned to convert all my VM's to NVMe passthru. No sense if it is slower.
Last edited: