Hello all.
I'm comparing NVME performance on GCP local ssds between FreeBSD 13.1 vs Debian 11 (default linux (5.10) install). With FreeBSD I've been unable to demonstrate performance beyond ~650k IOPS. GCP should support up to 2,400k IOPS with 24 local ssds. GCP uses a single controller for all local SSD namespaces.
For both tests I'm using fio with bs=4k, ioengine=posixaio, rw=randread and 16 jobs, one for each of 16 local ssds. I'm using posixaio because libaio is not supported on FreeBSD (fio limitation, or ?). Because posixaio effectively makes fio synchronous, I'm using numjobs=24, creating 24x16=384 processes to drive enough IO to the ssds.
Debian machine (n2-standard-8), running fio in this config gets 2,247k IOPS
FreeBSD machine (n2-standard-16) running fio in the config gets 459k IOPS. A bit more if I limit it to 1 process per job.
Am I running into a fio on FreeBSD limitation, or a NVME limitation on FreeBSD/GCP, or something else?
Thanks. Full fio file below.
- Michael
I'm comparing NVME performance on GCP local ssds between FreeBSD 13.1 vs Debian 11 (default linux (5.10) install). With FreeBSD I've been unable to demonstrate performance beyond ~650k IOPS. GCP should support up to 2,400k IOPS with 24 local ssds. GCP uses a single controller for all local SSD namespaces.
For both tests I'm using fio with bs=4k, ioengine=posixaio, rw=randread and 16 jobs, one for each of 16 local ssds. I'm using posixaio because libaio is not supported on FreeBSD (fio limitation, or ?). Because posixaio effectively makes fio synchronous, I'm using numjobs=24, creating 24x16=384 processes to drive enough IO to the ssds.
Debian machine (n2-standard-8), running fio in this config gets 2,247k IOPS
FreeBSD machine (n2-standard-16) running fio in the config gets 459k IOPS. A bit more if I limit it to 1 process per job.
Am I running into a fio on FreeBSD limitation, or a NVME limitation on FreeBSD/GCP, or something else?
Thanks. Full fio file below.
- Michael
Code:
[global]
bs=4K
size=100%
iodepth=64
ioengine=posixaio
group_reporting
time_based
runtime=60
numjobs=24
name=raw-randread
rw=randread
[job1]
filename=/dev/nvme0ns1
[job2]
filename=/dev/nvme0ns2
[job3]
filename=/dev/nvme0ns3
[job4]
filename=/dev/nvme0ns4
[job5]
filename=/dev/nvme0ns5
[job6]
filename=/dev/nvme0ns6
[job7]
filename=/dev/nvme0ns7
[job8]
filename=/dev/nvme0ns8
[job9]
filename=/dev/nvme0ns9
[job10]
filename=/dev/nvme0ns10
[job11]
filename=/dev/nvme0ns11
[job12]
filename=/dev/nvme0ns12
[job13]
filename=/dev/nvme0ns13
[job14]
filename=/dev/nvme0ns14
[job15]
filename=/dev/nvme0ns15
[job16]
filename=/dev/nvme0ns16