Last time I searched I couldn't find any conclusive information about how well ZFS native encryption performs against using GELI under ZFS.
So I ran some benchmarks on one of my systems to see what differences there are between GELI+ZFS and ZFS native encryption. The benchmark tool was bonnie++ (I have no idea if this is good or not) and the system was a Xeon-based server with 2 x 2.4GB 10K SAS drives set up as a ZFS mirror with compression on.
First the results, then the method used:
GELI + zstd compression
ZFS encryption + zstd compression
My interpretation is that performance is more-or-less the same (GELI+ZFS is a little better on most tests), except for sequential input, block where ZFS native performs way better, 2.5gb/sec vs 268mb/sec.
What's your take?
Method
The drives were set up as a ZFS mirror with
Test setup was:
So I ran some benchmarks on one of my systems to see what differences there are between GELI+ZFS and ZFS native encryption. The benchmark tool was bonnie++ (I have no idea if this is good or not) and the system was a Xeon-based server with 2 x 2.4GB 10K SAS drives set up as a ZFS mirror with compression on.
First the results, then the method used:
GELI + zstd compression
Code:
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 460k 98 1.4g 71 1.2g 97 33k 99 263m 99 +++++ +++
Latency 18466us 12598us 2473us 267ms 1042us 1703us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 30012.384455 61 +++++ +++ +++++ +++ +++++ +++
Latency 1307us 403us 95191us 1682us 349us 33046us
1.98,1.98,angola,1,1699114532,32G,,8192,5,460,98,1460356,71,1255071,97,33,99,269718,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,30012,61,+++++,+++,+++++,+++,+++++,+++,18466us,12598us,2473us,267ms,1042us,1703us,1307us,403us,95191us,1682us,349us,33046us
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 349k 99 1.1g 58 1.2g 94 34k 99 268m 99 +++++ +++
Latency 18933us 100ms 20308us 279ms 1042us 1942us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 28495.304888 60 +++++ +++ +++++ +++ +++++ +++
Latency 1305us 86us 106ms 1705us 545us 97350us
1.98,1.98,angola,1,1699114202,32G,,8192,5,349,99,1101700,58,1221264,94,34,99,274435,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,28495,60,+++++,+++,+++++,+++,+++++,+++,18933us,100ms,20308us,279ms,1042us,1942us,1305us,86us,106ms,1705us,545us,97350us
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 352k 99 1.1g 60 1.2g 98 34k 99 264m 99 +++++ +++
Latency 17731us 29464us 1047us 263ms 1043us 1652us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 29672.493293 60 +++++ +++ +++++ +++ +++++ +++
Latency 1288us 67us 74418us 1291us 353us 43078us
1.98,1.98,angola,1,1699113810,32G,,8192,5,352,99,1196099,60,1239384,98,34,99,270460,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,29672,60,+++++,+++,+++++,+++,+++++,+++,17731us,29464us,1047us,263ms,1043us,1652us,1288us,67us,74418us,1291us,353us,43078usZFS encryption + zstd compression
Code:
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 455k 97 1.3g 68 1.1g 91 952k 99 2.5g 99 +++++ +++
Latency 20519us 23159us 1188us 14053us 437us 697us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 16816.708167 32 +++++ +++ +++++ +++ 24278.597164 68
Latency 1522us 346us 470ms 1322us 530us 112ms
1.98,1.98,angola,1,1699115652,32G,,8192,5,455,97,1317361,68,1107614,91,952,99,2587478,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,16816,32,+++++,+++,+++++,+++,24278,68,20519us,23159us,1188us,14053us,437us,697us,1522us,346us,470ms,1322us,530us,112ms
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 366k 99 1.1g 61 1.1g 93 965k 99 2.5g 99 +++++ +++
Latency 18164us 24164us 8120us 12748us 1053us 741us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 18041.726993 36 +++++ +++ +++++ +++ 23917.933831 62
Latency 1513us 544us 413ms 1524us 352us 102ms
1.98,1.98,angola,1,1699115281,32G,,8192,5,366,99,1175057,61,1138814,93,965,99,2603273,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,18041,36,+++++,+++,+++++,+++,23917,62,18164us,24164us,8120us,12748us,1053us,741us,1513us,544us,413ms,1524us,352us,102ms
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
angola 32G 451k 99 1.2g 66 1.1g 93 960k 99 2.5g 99 +++++ +++
Latency 17827us 23987us 1991us 12407us 710us 759us
Version 1.98 ------Sequential Create------ --------Random Create--------
angola -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ 17661.225896 39 +++++ +++ +++++ +++ 25108.535957 51
Latency 1530us 349us 436ms 1425us 39us 133ms
1.98,1.98,angola,1,1699115370,32G,,8192,5,451,99,1290578,66,1133567,93,960,99,2583177,99,+++++,+++,16,,,,,+++++,+++,+++++,+++,17661,39,+++++,+++,+++++,+++,25108,51,17827us,23987us,1991us,12407us,710us,759us,1530us,349us,436ms,1425us,39us,133msMy interpretation is that performance is more-or-less the same (GELI+ZFS is a little better on most tests), except for sequential input, block where ZFS native performs way better, 2.5gb/sec vs 268mb/sec.
What's your take?
Method
The drives were set up as a ZFS mirror with
atime=off, compression=zstd, and vfs.zfs.min_auto_ashift: 12. bonnie++ was run 3 times consecutively for each configuration.Test setup was:
Code:
## Disk array setup
% gpart create -s gpt da0
% gpart create -s gpt da1
% gpart add -a 1m -s 2235G -t freebsd-zfs -l sas0 da0
% gpart add -a 1m -s 2235G -t freebsd-zfs -l sas1 da1
## GELI
% geli init -s 4096 /dev/gpt/sas0 /dev/gpt/sas1
% geli attach /dev/gpt/sas0 /dev/gpt/sas1
% sysctl vfs.zfs.min_auto_ashift
vfs.zfs.min_auto_ashift: 12
% zpool create storage mirror gpt/sas0.eli gpt/sas1.eli
% zfs set compression=zstd storage
% zfs set atime=off storage
% zfs create storage/test
## ZFS encryption
% sysctl vfs.zfs.min_auto_ashift
vfs.zfs.min_auto_ashift: 12
% zpool create storage mirror gpt/sas0 gpt/sas1
% zfs set compression=zstd storage
% zfs set atime=off storage
% zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase storage/test