I have two similar systems I'm testing as database hardware. Both consist of 8x Samsung 860 Pro SSD's attached to LSI9003-8i, 256GB ram, equal chassis/backplane. The only variance is one server is an Epyc 7371, and the other a Xeon Gold 6130. Some snippets to get the lay of the land, first the Xeon:
There are 8 configured SSD's in each all the same setup as the ones I listed in a ZFS raid 10 setup with 4 mirrored vdevs. I'm aware that GELI is setup with 4k aligned sectors vs the drives reported 512 bytes, it should be a non-factor for the behavior I'm seeing. Long and short is, despite the Epyc box reporting that it supports AES like the Xeon as well as GELI showing Crypto: hardware, when I extract a tar of my 100GB test database to each server the load of the Epyc box is significantly higher with top showing tons of CPU time on g_eli threads that the Xeon does not show when in hardware mode. When I disable the aesni and cryptodev modules on the Xeon I get the same behavior with high g_eli thread load which leads me to believe that depsite Epyc/GELI reporting they're using hardware AES it's actually still running in software mode. I've tried with both AES-XTS and AES-CBC modes for GELI with the same behavior. I'm not sure who the current maintainer of AESNI or GELI is hoping someone here can point me at them or they happen to be reading the forums.
Code:
xeon# kldstat
Id Refs Address Size Name
1 28 0xffffffff80200000 243cd00 kernel
2 1 0xffffffff8263e000 1ead0 geom_eli.ko
3 1 0xffffffff8265d000 79f0 cryptodev.ko
4 1 0xffffffff82665000 c7e0 aesni.ko
5 1 0xffffffff82822000 7078 ioat.ko
6 1 0xffffffff8282a000 1800 uhid.ko
7 1 0xffffffff8282c000 23a8 ums.ko
8 1 0xffffffff8282f000 247e20 zfs.ko
9 1 0xffffffff82a77000 7628 opensolaris.ko
xeon# geli list da0.eli
Geom name: da0.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
UsedKey: 0
Flags: NONE
KeysAllocated: 120
KeysTotal: 120
Providers:
1. Name: da0.eli
Mediasize: 512110186496 (477G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: da0
Mediasize: 512110190592 (477G)
Sectorsize: 512
Mode: r1w1e1
xeon# grep -i aesni /var/run/dmesg.boot
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
Code:
epyc# kldstat
Id Refs Address Size Name
1 34 0xffffffff80200000 2448ef8 kernel
2 1 0xffffffff82649000 e1d8 aesni.ko
3 1 0xffffffff82659000 1e8a0 geom_eli.ko
4 1 0xffffffff82678000 90b8 cryptodev.ko
5 1 0xffffffff82822000 2668 intpm.ko
6 1 0xffffffff82825000 b50 smbus.ko
7 1 0xffffffff82826000 18a0 uhid.ko
8 1 0xffffffff82828000 2928 ums.ko
9 1 0xffffffff8282b000 acf mac_ntpd.ko
10 1 0xffffffff8282c000 2472e0 zfs.ko
11 1 0xffffffff82a74000 7628 opensolaris.ko
epyc# geli list da0.eli
Geom name: da0.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
UsedKey: 0
Flags: NONE
KeysAllocated: 120
KeysTotal: 120
Providers:
1. Name: da0.eli
Mediasize: 512110186496 (477G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: da0
Mediasize: 512110190592 (477G)
Sectorsize: 512
Mode: r1w1e1
epyc# grep -i aesni /var/run/dmesg.boot
Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256> on motherboardThere are 8 configured SSD's in each all the same setup as the ones I listed in a ZFS raid 10 setup with 4 mirrored vdevs. I'm aware that GELI is setup with 4k aligned sectors vs the drives reported 512 bytes, it should be a non-factor for the behavior I'm seeing. Long and short is, despite the Epyc box reporting that it supports AES like the Xeon as well as GELI showing Crypto: hardware, when I extract a tar of my 100GB test database to each server the load of the Epyc box is significantly higher with top showing tons of CPU time on g_eli threads that the Xeon does not show when in hardware mode. When I disable the aesni and cryptodev modules on the Xeon I get the same behavior with high g_eli thread load which leads me to believe that depsite Epyc/GELI reporting they're using hardware AES it's actually still running in software mode. I've tried with both AES-XTS and AES-CBC modes for GELI with the same behavior. I'm not sure who the current maintainer of AESNI or GELI is hoping someone here can point me at them or they happen to be reading the forums.
