AWS Disk I/O Performance: xbd vs nvd

gcomeau

New Member


Messages: 9

This is a reboot of Thread freebsd-11-2-vs-12-1-vast-difference-in-performance-profile.74671, which has some useful but non-essential context. I am waiting/hoping to find a better home for my question.

I observe a significant IO performance degradation on the latest AWS instance types which I think has to do with I/O command overhead on the nvd driver.
The test setup is an instance of FreeBSD 12.1-STABLE-amd64-2019-10-31 (ami-0090ffd64673f5607). I simply attach a 100 GB EBS disk on the instance and run diskinfo -c -S -i -t -w on an m4.large. Stop the instance, change type to m5.large and repeat. For reasons that I think relate to this post from cpercival, the m5 mounts EBS during nvd and the m4 as xbd. The results are higher I/O command overhead and higher I/O throughput which is fine unless you run a high friction ACID-compliant database.

Full results listed below. The salient points are

awsm4m5.png


where I believe the extra command overhead accounts for the mysql slave performance degradation observed in the original post.

The question now is - is there anything one can do to mitigate I/O command overhead on nvd, even at the expense of throughput? Or that's just life in the cloud?

Full report on m4
Code:
root@freebsd:/usr/home/ec2-user # uname -a && diskinfo -c -S -i -t -w /dev/xbd1 
FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r354199 GENERIC  amd64
/dev/xbd1
    512             # sectorsize
    107374182400    # mediasize in bytes (100G)
    209715200       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
                    # Disk descr.
                    # Disk ident.
    No              # TRIM/UNMAP support
    Unknown         # Rotation rate in RPM

I/O command overhead:
    time to read 10MB block      0.032032 sec    =    0.002 msec/sector
    time to read 20480 sectors   5.693211 sec    =    0.278 msec/sector
    calculated command overhead            =    0.276 msec/sector

Seek times:
    Full stroke:      250 iter in   0.067383 sec =    0.270 msec
    Half stroke:      250 iter in   0.062631 sec =    0.251 msec
    Quarter stroke:      500 iter in   0.119734 sec =    0.239 msec
    Short forward:      400 iter in   0.083028 sec =    0.208 msec
    Short backward:      400 iter in   0.098988 sec =    0.247 msec
    Seq outer:     2048 iter in   0.384474 sec =    0.188 msec
    Seq inner:     2048 iter in   0.384832 sec =    0.188 msec

Transfer rates:
    outside:       102400 kbytes in   0.863108 sec =   118641 kbytes/sec
    middle:        102400 kbytes in   0.863002 sec =   118656 kbytes/sec
    inside:        102400 kbytes in   0.863020 sec =   118653 kbytes/sec

Asynchronous random reads:
    sectorsize:     12366 ops in    3.041734 sec =     4065 IOPS
    4 kbytes:       12367 ops in    3.041852 sec =     4066 IOPS
    32 kbytes:       6998 ops in    3.074209 sec =     2276 IOPS
    128 kbytes:      1845 ops in    3.296433 sec =      560 IOPS

Synchronous random writes:
     0.5 kbytes:    718.4 usec/IO =      0.7 Mbytes/s
       1 kbytes:    716.4 usec/IO =      1.4 Mbytes/s
       2 kbytes:    749.1 usec/IO =      2.6 Mbytes/s
       4 kbytes:    778.8 usec/IO =      5.0 Mbytes/s
       8 kbytes:    771.7 usec/IO =     10.1 Mbytes/s
      16 kbytes:    802.6 usec/IO =     19.5 Mbytes/s
      32 kbytes:    898.2 usec/IO =     34.8 Mbytes/s
      64 kbytes:   1090.2 usec/IO =     57.3 Mbytes/s
     128 kbytes:   1405.2 usec/IO =     89.0 Mbytes/s
     256 kbytes:   2658.8 usec/IO =     94.0 Mbytes/s
     512 kbytes:   5317.6 usec/IO =     94.0 Mbytes/s
    1024 kbytes:  14630.9 usec/IO =     68.3 Mbytes/s
    2048 kbytes:  33256.7 usec/IO =     60.1 Mbytes/s
    4096 kbytes:  70500.6 usec/IO =     56.7 Mbytes/s
    8192 kbytes: 145011.3 usec/IO =     55.2 Mbytes/s
root@freebsd:/usr/home/ec2-user # poweroff
Full report on m5
Code:
root@freebsd:/usr/home/ec2-user #  uname -a && diskinfo -c -S -i -t -w /dev/nvd1
FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r354199 GENERIC  amd64
/dev/nvd1
    512             # sectorsize
    107374182400    # mediasize in bytes (100G)
    209715200       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    Amazon Elastic Block Store    # Disk descr.
    vol06c33140f6e0c82fb    # Disk ident.
    No              # TRIM/UNMAP support
    0               # Rotation rate in RPM

I/O command overhead:
    time to read 10MB block      0.047512 sec    =    0.002 msec/sector
    time to read 20480 sectors   7.932988 sec    =    0.387 msec/sector
    calculated command overhead            =    0.385 msec/sector

Seek times:
    Full stroke:      250 iter in   0.097874 sec =    0.391 msec
    Half stroke:      250 iter in   0.096626 sec =    0.387 msec
    Quarter stroke:      500 iter in   0.191641 sec =    0.383 msec
    Short forward:      400 iter in   0.164443 sec =    0.411 msec
    Short backward:      400 iter in   0.168033 sec =    0.420 msec
    Seq outer:     2048 iter in   0.781036 sec =    0.381 msec
    Seq inner:     2048 iter in   0.807298 sec =    0.394 msec

Transfer rates:
    outside:       102400 kbytes in   0.457210 sec =   223967 kbytes/sec
    middle:        102400 kbytes in   0.474336 sec =   215881 kbytes/sec
    inside:        102400 kbytes in   0.438660 sec =   233438 kbytes/sec

Asynchronous random reads:
    sectorsize:     12126 ops in    3.042614 sec =     3985 IOPS
    4 kbytes:       12126 ops in    3.042399 sec =     3986 IOPS
    32 kbytes:      12127 ops in    3.042503 sec =     3986 IOPS
    128 kbytes:      4227 ops in    3.124038 sec =     1353 IOPS

Synchronous random writes:
     0.5 kbytes:    858.7 usec/IO =      0.6 Mbytes/s
       1 kbytes:    885.5 usec/IO =      1.1 Mbytes/s
       2 kbytes:    961.9 usec/IO =      2.0 Mbytes/s
       4 kbytes:    981.0 usec/IO =      4.0 Mbytes/s
       8 kbytes:    995.7 usec/IO =      7.8 Mbytes/s
      16 kbytes:   1010.6 usec/IO =     15.5 Mbytes/s
      32 kbytes:   1068.9 usec/IO =     29.2 Mbytes/s
      64 kbytes:   1212.9 usec/IO =     51.5 Mbytes/s
     128 kbytes:   1568.7 usec/IO =     79.7 Mbytes/s
     256 kbytes:   2212.9 usec/IO =    113.0 Mbytes/s
     512 kbytes:   2648.9 usec/IO =    188.8 Mbytes/s
    1024 kbytes:   4253.4 usec/IO =    235.1 Mbytes/s
    2048 kbytes:  11618.8 usec/IO =    172.1 Mbytes/s
    4096 kbytes:  27223.3 usec/IO =    146.9 Mbytes/s
    8192 kbytes:  58436.4 usec/IO =    136.9 Mbytes/s
root@freebsd:/usr/home/ec2-user #  
root@freebsd:/usr/home/ec2-user # poweroff
 
Last edited by a moderator:
OP
G

gcomeau

New Member


Messages: 9

Sidebar - if anyone knows of a better venue for FreeBSD on AWS issues, please let me know. The response rate on this forum is generally good.
 

Phishfry

Son of Beastie

Reaction score: 1,779
Messages: 4,838

Have you tried exposing the NVMe to the nda(4) driver instead of nvd?
Add to /boot/loader.conf
hw.nvme.use_nvd=0
 

Phishfry

Son of Beastie

Reaction score: 1,779
Messages: 4,838

I wanted to point out my thread with some bhyve numbers:

As you can see there is a significant drop from raw host numbers.
From ~1900MB/s down to ~600MB/s when passed thru.
I would not anticipate any better from AWS.
 
OP
G

gcomeau

New Member


Messages: 9

Thanks - I checked the diskinfo numbers which look better. Trying as a mariadb slave & will post an update once I have data.
 
OP
G

gcomeau

New Member


Messages: 9

First impression was incorrect. Actually looks similar to nvd=1. I have actually run it as a slave for a few days and it doesn't track any better.

Back to the hypothesis that something about using the latest AWS t4/m5 tech that maps EBS to NVMe devices causes a significant slowdown due to command overhead. The I/O command overhead is 0.437 msec/sector.

Code:
vm.pmap.pti=0
hw.nvme.use_nvd=0
root@freebsd:/usr/home/ec2-user # uname -a && diskinfo -c -t -i -S -w /dev/nda1
FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r354199 GENERIC  amd64
/dev/nda1
    512             # sectorsize
    107374182400    # mediasize in bytes (100G)
    209715200       # mediasize in sectors
    512             # stripesize
    0               # stripeoffset
    Amazon Elastic Block Store                 # Disk descr.
    vol06c33140f6e0c82f    # Disk ident.
    No              # TRIM/UNMAP support
    0               # Rotation rate in RPM

I/O command overhead:
    time to read 10MB block      0.084747 sec    =    0.004 msec/sector
    time to read 20480 sectors   9.027117 sec    =    0.441 msec/sector
    calculated command overhead            =    0.437 msec/sector

Seek times:
    Full stroke:      250 iter in   0.116940 sec =    0.468 msec
    Half stroke:      250 iter in   0.121797 sec =    0.487 msec
    Quarter stroke:      500 iter in   0.235433 sec =    0.471 msec
    Short forward:      400 iter in   0.179722 sec =    0.449 msec
    Short backward:      400 iter in   0.188964 sec =    0.472 msec
    Seq outer:     2048 iter in   0.899846 sec =    0.439 msec
    Seq inner:     2048 iter in   0.904801 sec =    0.442 msec

Transfer rates:
    outside:       102400 kbytes in   0.623726 sec =   164175 kbytes/sec
    middle:        102400 kbytes in   0.663239 sec =   154394 kbytes/sec
    inside:        102400 kbytes in   0.624691 sec =   163921 kbytes/sec

Asynchronous random reads:
    sectorsize:     12126 ops in    3.042450 sec =     3986 IOPS
    4 kbytes:       12126 ops in    3.042528 sec =     3986 IOPS
    32 kbytes:      12127 ops in    3.042503 sec =     3986 IOPS
    128 kbytes:      4227 ops in    3.123997 sec =     1353 IOPS

Synchronous random writes:
     0.5 kbytes:    685.8 usec/IO =      0.7 Mbytes/s
       1 kbytes:    697.4 usec/IO =      1.4 Mbytes/s
       2 kbytes:    709.0 usec/IO =      2.8 Mbytes/s
       4 kbytes:    737.3 usec/IO =      5.3 Mbytes/s
       8 kbytes:    779.9 usec/IO =     10.0 Mbytes/s
      16 kbytes:    827.7 usec/IO =     18.9 Mbytes/s
      32 kbytes:    885.2 usec/IO =     35.3 Mbytes/s
      64 kbytes:   1047.4 usec/IO =     59.7 Mbytes/s
     128 kbytes:   1402.8 usec/IO =     89.1 Mbytes/s
     256 kbytes:   1795.0 usec/IO =    139.3 Mbytes/s
     512 kbytes:   2580.9 usec/IO =    193.7 Mbytes/s
    1024 kbytes:   4113.0 usec/IO =    243.1 Mbytes/s
    2048 kbytes:  11619.8 usec/IO =    172.1 Mbytes/s
    4096 kbytes:  27224.4 usec/IO =    146.9 Mbytes/s
    8192 kbytes:  58438.9 usec/IO =    136.9 Mbytes/s
 
Last edited by a moderator:
Top