This is a reboot of Thread freebsd-11-2-vs-12-1-vast-difference-in-performance-profile.74671, which has some useful but non-essential context. I am waiting/hoping to find a better home for my question.
I observe a significant IO performance degradation on the latest AWS instance types which I think has to do with I/O command overhead on the nvd driver.
The test setup is an instance of FreeBSD 12.1-STABLE-amd64-2019-10-31 (ami-0090ffd64673f5607). I simply attach a 100 GB EBS disk on the instance and run
Full results listed below. The salient points are
where I believe the extra command overhead accounts for the mysql slave performance degradation observed in the original post.
The question now is - is there anything one can do to mitigate I/O command overhead on nvd, even at the expense of throughput? Or that's just life in the cloud?
Full report on m4
Full report on m5
I observe a significant IO performance degradation on the latest AWS instance types which I think has to do with I/O command overhead on the nvd driver.
The test setup is an instance of FreeBSD 12.1-STABLE-amd64-2019-10-31 (ami-0090ffd64673f5607). I simply attach a 100 GB EBS disk on the instance and run
diskinfo -c -S -i -t -w
on an m4.large. Stop the instance, change type to m5.large and repeat. For reasons that I think relate to this post from cpercival, the m5 mounts EBS during nvd and the m4 as xbd. The results are higher I/O command overhead and higher I/O throughput which is fine unless you run a high friction ACID-compliant database.Full results listed below. The salient points are
where I believe the extra command overhead accounts for the mysql slave performance degradation observed in the original post.
The question now is - is there anything one can do to mitigate I/O command overhead on nvd, even at the expense of throughput? Or that's just life in the cloud?
Full report on m4
Code:
root@freebsd:/usr/home/ec2-user # uname -a && diskinfo -c -S -i -t -w /dev/xbd1
FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r354199 GENERIC amd64
/dev/xbd1
512 # sectorsize
107374182400 # mediasize in bytes (100G)
209715200 # mediasize in sectors
0 # stripesize
0 # stripeoffset
# Disk descr.
# Disk ident.
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
I/O command overhead:
time to read 10MB block 0.032032 sec = 0.002 msec/sector
time to read 20480 sectors 5.693211 sec = 0.278 msec/sector
calculated command overhead = 0.276 msec/sector
Seek times:
Full stroke: 250 iter in 0.067383 sec = 0.270 msec
Half stroke: 250 iter in 0.062631 sec = 0.251 msec
Quarter stroke: 500 iter in 0.119734 sec = 0.239 msec
Short forward: 400 iter in 0.083028 sec = 0.208 msec
Short backward: 400 iter in 0.098988 sec = 0.247 msec
Seq outer: 2048 iter in 0.384474 sec = 0.188 msec
Seq inner: 2048 iter in 0.384832 sec = 0.188 msec
Transfer rates:
outside: 102400 kbytes in 0.863108 sec = 118641 kbytes/sec
middle: 102400 kbytes in 0.863002 sec = 118656 kbytes/sec
inside: 102400 kbytes in 0.863020 sec = 118653 kbytes/sec
Asynchronous random reads:
sectorsize: 12366 ops in 3.041734 sec = 4065 IOPS
4 kbytes: 12367 ops in 3.041852 sec = 4066 IOPS
32 kbytes: 6998 ops in 3.074209 sec = 2276 IOPS
128 kbytes: 1845 ops in 3.296433 sec = 560 IOPS
Synchronous random writes:
0.5 kbytes: 718.4 usec/IO = 0.7 Mbytes/s
1 kbytes: 716.4 usec/IO = 1.4 Mbytes/s
2 kbytes: 749.1 usec/IO = 2.6 Mbytes/s
4 kbytes: 778.8 usec/IO = 5.0 Mbytes/s
8 kbytes: 771.7 usec/IO = 10.1 Mbytes/s
16 kbytes: 802.6 usec/IO = 19.5 Mbytes/s
32 kbytes: 898.2 usec/IO = 34.8 Mbytes/s
64 kbytes: 1090.2 usec/IO = 57.3 Mbytes/s
128 kbytes: 1405.2 usec/IO = 89.0 Mbytes/s
256 kbytes: 2658.8 usec/IO = 94.0 Mbytes/s
512 kbytes: 5317.6 usec/IO = 94.0 Mbytes/s
1024 kbytes: 14630.9 usec/IO = 68.3 Mbytes/s
2048 kbytes: 33256.7 usec/IO = 60.1 Mbytes/s
4096 kbytes: 70500.6 usec/IO = 56.7 Mbytes/s
8192 kbytes: 145011.3 usec/IO = 55.2 Mbytes/s
root@freebsd:/usr/home/ec2-user # poweroff
Full report on m5
Code:
root@freebsd:/usr/home/ec2-user # uname -a && diskinfo -c -S -i -t -w /dev/nvd1
FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r354199 GENERIC amd64
/dev/nvd1
512 # sectorsize
107374182400 # mediasize in bytes (100G)
209715200 # mediasize in sectors
0 # stripesize
0 # stripeoffset
Amazon Elastic Block Store # Disk descr.
vol06c33140f6e0c82fb # Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM
I/O command overhead:
time to read 10MB block 0.047512 sec = 0.002 msec/sector
time to read 20480 sectors 7.932988 sec = 0.387 msec/sector
calculated command overhead = 0.385 msec/sector
Seek times:
Full stroke: 250 iter in 0.097874 sec = 0.391 msec
Half stroke: 250 iter in 0.096626 sec = 0.387 msec
Quarter stroke: 500 iter in 0.191641 sec = 0.383 msec
Short forward: 400 iter in 0.164443 sec = 0.411 msec
Short backward: 400 iter in 0.168033 sec = 0.420 msec
Seq outer: 2048 iter in 0.781036 sec = 0.381 msec
Seq inner: 2048 iter in 0.807298 sec = 0.394 msec
Transfer rates:
outside: 102400 kbytes in 0.457210 sec = 223967 kbytes/sec
middle: 102400 kbytes in 0.474336 sec = 215881 kbytes/sec
inside: 102400 kbytes in 0.438660 sec = 233438 kbytes/sec
Asynchronous random reads:
sectorsize: 12126 ops in 3.042614 sec = 3985 IOPS
4 kbytes: 12126 ops in 3.042399 sec = 3986 IOPS
32 kbytes: 12127 ops in 3.042503 sec = 3986 IOPS
128 kbytes: 4227 ops in 3.124038 sec = 1353 IOPS
Synchronous random writes:
0.5 kbytes: 858.7 usec/IO = 0.6 Mbytes/s
1 kbytes: 885.5 usec/IO = 1.1 Mbytes/s
2 kbytes: 961.9 usec/IO = 2.0 Mbytes/s
4 kbytes: 981.0 usec/IO = 4.0 Mbytes/s
8 kbytes: 995.7 usec/IO = 7.8 Mbytes/s
16 kbytes: 1010.6 usec/IO = 15.5 Mbytes/s
32 kbytes: 1068.9 usec/IO = 29.2 Mbytes/s
64 kbytes: 1212.9 usec/IO = 51.5 Mbytes/s
128 kbytes: 1568.7 usec/IO = 79.7 Mbytes/s
256 kbytes: 2212.9 usec/IO = 113.0 Mbytes/s
512 kbytes: 2648.9 usec/IO = 188.8 Mbytes/s
1024 kbytes: 4253.4 usec/IO = 235.1 Mbytes/s
2048 kbytes: 11618.8 usec/IO = 172.1 Mbytes/s
4096 kbytes: 27223.3 usec/IO = 146.9 Mbytes/s
8192 kbytes: 58436.4 usec/IO = 136.9 Mbytes/s
root@freebsd:/usr/home/ec2-user #
root@freebsd:/usr/home/ec2-user # poweroff
Last edited by a moderator: