Hi
I am benchmarking ZFS in preparing to use FreeNAS for ESXi datastore. The problem I found is the IOPS of the pool is much lower then the SLOG, even with a small test size
I was expecting 4K write IOPS to be close to 40K according to
Details of my pool
Key words: 4 mirrored vdevs, low utilization and fragmentation, recordsize=16K, sync=always, compression off
My hardware:
Dell R620 as head unit:
Dual E5 2690 v2
128GB RAM
2 s3500 80G, boot drive
Dell network card with 2 x540 2 i350
LSI 9207 8e
P3700, slog
NetAPP DS4243 as DAS:
24 3.5 Bay
IOM3
12 4TB HGST NL SAS drives HUS726040AL5210
I am benchmarking ZFS in preparing to use FreeNAS for ESXi datastore. The problem I found is the IOPS of the pool is much lower then the SLOG, even with a small test size
Code:
root@freenas:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
4096 # sectorsize
42999996416 # mediasize in bytes (40G)
10498046 # mediasize in sectors
131072 # stripesize
0 # stripeoffset
INTEL SSDPEDMD400G4 # Disk descr.
CVFT7112001A400LGN # Disk ident.
Synchronous random writes:
4 kbytes: 22.1 usec/IO = 176.7 Mbytes/s
8 kbytes: 24.2 usec/IO = 322.4 Mbytes/s
16 kbytes: 29.3 usec/IO = 533.3 Mbytes/s
32 kbytes: 39.4 usec/IO = 793.8 Mbytes/s
64 kbytes: 72.4 usec/IO = 862.7 Mbytes/s
128 kbytes: 147.8 usec/IO = 845.8 Mbytes/s
256 kbytes: 250.1 usec/IO = 999.5 Mbytes/s
512 kbytes: 490.6 usec/IO = 1019.2 Mbytes/s
1024 kbytes: 961.4 usec/IO = 1040.1 Mbytes/s
2048 kbytes: 1926.7 usec/IO = 1038.1 Mbytes/s
4096 kbytes: 3806.6 usec/IO = 1050.8 Mbytes/s
8192 kbytes: 7432.6 usec/IO = 1076.3 Mbytes/s
root@freenas:/mnt/m8 # iozone -a -s 512M -O
Iozone: Performance Test of File I/O
Version $Revision: 3.457 $
Compiled for 64 bit mode.
Build: freebsd
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England, Vikentsi Lapa,
Alexey Skidanov.
Run began: Tue Oct 9 09:50:10 2018
Auto Mode
File size set to 524288 kB
OPS Mode. Output is in operations per second.
Command line used: iozone -a -s 512M -O
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
524288 4 9281 9510 162554 340426 250513 8166 135176 10920 160345 10034 9485 147176 315245
524288 8 8563 8531 110181 260793 214246 7899 98585 10013 97223 8552 8609 103698 155445
524288 16 6614 6526 83307 87849 152698 7025 76399 8758 76104 6624 6651 71240 77383
524288 32 4552 4337 52874 62135 107487 4575 52359 5898 51267 4313 4279 45693 48314
524288 64 2717 2703 29693 30646 34548 2801 29705 3497 29035 2784 2717 22968 25298
524288 128 1666 1529 16272 16223 16586 1698 16168 1869 15943 1539 1565 11979 12701
524288 256 1013 1006 8422 8435 16472 1186 8359 1284 8376 1032 1019 6034 6038
524288 512 587 593 4388 4411 7132 724 4493 1424 4473 611 609 3048 3095
524288 1024 318 315 2253 2245 2348 466 2237 427 2136 326 327 1538 1571
524288 2048 173 168 1136 1128 1741 188 1174 241 1150 170 167 753 801
524288 4096 114 85 559 583 637 90 581 128 642 88 85 398 452
524288 8192 43 42 248 261 282 42 285 57 274 43 43 148 168
524288 16384 21 21 109 128 136 23 109 26 108 26 21 58 71 diskinfo -wS results, however it's only 10K. Is this how ZFS work or something is wrong?Details of my pool
Code:
root@freenas:/mnt/m8 # zpool list -v m8
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
m8 14.5T 447G 14.1T - 0% 3% 1.00x ONLINE /mnt
mirror 3.62T 109G 3.52T - 0% 2%
gptid/1956925c-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
gptid/1a38d15c-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
mirror 3.62T 113G 3.51T - 0% 3%
gptid/1b4b8875-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
gptid/18050e27-c9f1-11e8-b164-ecf4bbcb7e90 - - - - - -
mirror 3.62T 113G 3.51T - 0% 3%
gptid/1d5c5df5-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
gptid/1e5ee446-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
mirror 3.62T 111G 3.52T - 0% 2%
gptid/3cdb7a3e-c9f1-11e8-b164-ecf4bbcb7e90 - - - - - -
gptid/6cbeef4b-c9e5-11e8-b164-ecf4bbcb7e90 - - - - - -
log - - - - - -
nvd0 40G 128K 40.0G - 0% 0%
root@freenas:/mnt/m8 # zfs get all m8
NAME PROPERTY VALUE SOURCE
m8 type filesystem -
m8 creation Sat Oct 6 20:57 2018 -
m8 used 447G -
m8 available 13.6T -
m8 referenced 447G -
m8 compressratio 1.00x -
m8 mounted yes -
m8 quota none local
m8 reservation none local
m8 recordsize 16K local
m8 mountpoint /mnt/m8 default
m8 sharenfs off default
m8 checksum on default
m8 compression off local
m8 atime on default
m8 devices on default
m8 exec on default
m8 setuid on default
m8 readonly off default
m8 jailed off default
m8 snapdir hidden default
m8 aclmode passthrough local
m8 aclinherit passthrough local
m8 canmount on default
m8 xattr off temporary
m8 copies 1 default
m8 version 5 -
m8 utf8only off -
m8 normalization none -
m8 casesensitivity sensitive -
m8 vscan off default
m8 nbmand off default
m8 refquota none local
m8 refreservation none local
m8 primarycache all default
m8 secondarycache all default
m8 usedbysnapshots 0 -
m8 usedbydataset 447G -
m8 usedbychildren 37.5M -
m8 usedbyrefreservation 0 -
m8 logbias latency default
m8 dedup off default
m8 mlslabel -
m8 sync always local
m8 refcompressratio 1.00x -
m8 written 447G -
m8 logicalused 447G -
m8 logicalreferenced 447G -
m8 volmode default default
m8 filesystem_limit none default
m8 snapshot_limit none default
m8 filesystem_count none default
m8 snapshot_count none default
m8 redundant_metadata all defaultMy hardware:
Dell R620 as head unit:
Dual E5 2690 v2
128GB RAM
2 s3500 80G, boot drive
Dell network card with 2 x540 2 i350
LSI 9207 8e
P3700, slog
NetAPP DS4243 as DAS:
24 3.5 Bay
IOM3
12 4TB HGST NL SAS drives HUS726040AL5210