NFS write performance with mirrored ZIL

OP
OP
Sebulon

Sebulon

Aspiring Daemon

Reaction score: 128
Messages: 709

@jrm

The results always fluctuate a little, so what I do is to run all tests three times and then post the middle as the score. So your results where more or less the same.

Your collegues benchmark is rather invalid since it was performed asynchronosly, but good to see it as a "best case", only reflected the theoretical speed of the network as such.

/Sebulon
 

jrm@

Daemon
Developer

Reaction score: 473
Messages: 1,205

I actually did run the tests several times and all the results were with a fraction of a MB/s and the speeds were consistently slower after the 4K alignment of the SSDs with the ZIL and L2ARC. Perhaps a confounding variable here is that the pool had almost nothing on it when I ran the benchmarks before and there was about 200GB when I ran it after updating the alignment.
 
OP
OP
Sebulon

Sebulon

Aspiring Daemon

Reaction score: 128
Messages: 709

Time for an update on a new addition to our ZIL-family, the Intel DC S3700 200GB MLC!

Code:
[B][U]HW[/U][/B]
1x  Supermicro X8SIL-F
2x  Supermicro AOC-USAS2-L8i
2x  Supermicro CSE-M35T-1B
1x  Intel Xeon X3470 2.93GHz
4x  8GB 1333MHZ DDR3 ECC RDIMM
10x ST4000DM000-1F2168
2x  Intel S3700 200GB
1x  OCZ Deneva 2 R-series 200GB
1x  OCZ Vertex 3 240GB

[B][U]SW[/U][/B]
[CMD="#"]uname -a[/CMD]
FreeBSD server 9.1-RELEASE-p7 FreeBSD 9.1-RELEASE-p7 #0 r255487M: Thu Sep 12 08:26:57 CEST 2013     root@server:/usr/obj/usr/src/sys/server  amd64
[CMD="#"]zpool get version pool2[/CMD]
NAME   PROPERTY  VALUE    SOURCE
pool1  version   28       default
[CMD="#"]zpool status[/CMD]
  pool: pool1
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Sep 29 03:30:10 2013
config:

	NAME          STATE     READ WRITE CKSUM
	pool1         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    gpt/usb1  ONLINE       0     0     0
	    gpt/usb2  ONLINE       0     0     0

errors: No known data errors

  pool: pool2
 state: ONLINE
  scan: resilvered 0 in 0h40m with 0 errors on Tue Oct  1 11:25:30 2013
config:

	NAME             STATE     READ WRITE CKSUM
	pool2            ONLINE       0     0     0
	  raidz2-0       ONLINE       0     0     0
	    gpt/rack1-1  ONLINE       0     0     0
	    gpt/rack1-2  ONLINE       0     0     0
	    gpt/rack1-3  ONLINE       0     0     0
	    gpt/rack1-4  ONLINE       0     0     0
	    gpt/rack1-5  ONLINE       0     0     0
	    gpt/rack2-1  ONLINE       0     0     0
	    gpt/rack2-2  ONLINE       0     0     0
	    gpt/rack2-3  ONLINE       0     0     0
	    gpt/rack2-4  ONLINE       0     0     0
	    gpt/rack2-5  ONLINE       0     0     0
	logs
	  mirror-1       ONLINE       0     0     0
	    gpt/log1     ONLINE       0     0     0
	    gpt/log2     ONLINE       0     0     0
	cache
	  gpt/cache1     ONLINE       0     0     0
	  gpt/cache2     ONLINE       0     0     0
[CMD="#"]zdb | grep ashift[/CMD]
            ashift: 12
            ashift: 12
            ashift: 12
[CMD="#"]camcontrol devlist[/CMD]
<ATA ST4000DM000-1F21 CC52>        at scbus0 target 0 lun 0 (da0,pass0)
<ATA ST4000DM000-1F21 CC52>        at scbus0 target 1 lun 0 (da1,pass1)
<ATA INTEL SSDSC2BA20 0265>        at scbus0 target 3 lun 0 (da2,pass2)
<ATA ST4000DM000-1F21 CC52>        at scbus0 target 4 lun 0 (da3,pass3)
<ATA ST4000DM000-1F21 CC52>        at scbus0 target 5 lun 0 (da4,pass4)
<ATA ST4000DM000-1F21 CC52>        at scbus0 target 6 lun 0 (da5,pass5)
<ATA INTEL SSDSC2BA20 0265>        at scbus0 target 7 lun 0 (da6,pass6)
<ATA ST4000DM000-1F21 CC52>        at scbus1 target 0 lun 0 (da7,pass7)
<ATA ST4000DM000-1F21 CC52>        at scbus1 target 1 lun 0 (da8,pass8)
<ATA ST4000DM000-1F21 CC52>        at scbus1 target 2 lun 0 (da9,pass9)
<ATA OCZ-VERTEX3 2.15>             at scbus1 target 3 lun 0 (da10,pass10)
<ATA ST4000DM000-1F21 CC52>        at scbus1 target 4 lun 0 (da11,pass11)
<ATA ST4000DM000-1F21 CC52>        at scbus1 target 5 lun 0 (da12,pass12)
<ATA D2RSTK251M11-020 E>           at scbus1 target 6 lun 0 (da13,pass13)
<Kingston DataTraveler SE9 PMAP>   at scbus6 target 0 lun 0 (da14,pass14)
<Kingston DataTraveler SE9 PMAP>   at scbus7 target 0 lun 0 (pass15,da15)
[CMD="#"]gpart show -l[/CMD]
=>        34  7814037101  da0  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack2-1  (3.7T)
  7814037128           7       - free -  (3.5k)

=>        34  7814037101  da1  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack2-2  (3.7T)
  7814037128           7       - free -  (3.5k)

=>       34  390721901  da2  GPT  (186G)
         34       2014       - free -  (1M)
       2048  100663296    1  log2  (48G)
  100665344   33554432    2  swap2  (16G)
  134219776  256502159       - free -  (122G)

=>        34  7814037101  da3  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack2-3  (3.7T)
  7814037128           7       - free -  (3.5k)

=>        34  7814037101  da4  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack2-4  (3.7T)
  7814037128           7       - free -  (3.5k)

=>        34  7814037101  da5  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack2-5  (3.7T)
  7814037128           7       - free -  (3.5k)

=>       34  390721901  da6  GPT  (186G)
         34       2014       - free -  (1M)
       2048  100663296    1  log1  (48G)
  100665344   33554432    2  swap1  (16G)
  134219776  256502159       - free -  (122G)

=>        34  7814037101  da7  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack1-3  (3.7T)
  7814037128           7       - free -  (3.5k)

=>        34  7814037101  da8  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack1-4  (3.7T)
  7814037128           7       - free -  (3.5k)

=>        34  7814037101  da9  GPT  (3.7T)
          34        2014       - free -  (1M)
        2048  7814035080    1  rack1-5  (3.7T)
  7814037128           7       - free -  (3.5k)

=>       34  468862061  da10  GPT  (223G)
         34       2014        - free -  (1M)
       2048  468860040     1  cache1  (223G)
  468862088          7        - free -  (3.5k)

=>        34  7814037101  da11  GPT  (3.7T)
          34        2014        - free -  (1M)
        2048  7814035080     1  rack1-1  (3.7T)
  7814037128           7        - free -  (3.5k)

=>        34  7814037101  da12  GPT  (3.7T)
          34        2014        - free -  (1M)
        2048  7814035080     1  rack1-2  (3.7T)
  7814037128           7        - free -  (3.5k)

=>       34  390721901  da13  GPT  (186G)
         34       2014        - free -  (1M)
       2048  390719880     1  cache2  (186G)
  390721928          7        - free -  (3.5k)

=>      34  15356093  da14  GPT  (7.3G)
        34       128     1  (null)  (64k)
       162      1886        - free -  (943k)
      2048  14680064     2  usb1  (7.0G)
  14682112    674015        - free -  (329M)

=>      34  15470525  da15  GPT  (7.4G)
        34       128     1  (null)  (64k)
       162      1886        - free -  (943k)
      2048  14680064     2  usb2  (7.0G)
  14682112    788447        - free -  (385M)
Code:
[B][U]LOCAL WRITES[/U][/B]
128k)  295MB/s
4k)    58MB/s
Code:
Connection speed test from a KVM virtual machine running FreeBSD-9.2-RELEASE.
[CMD="#"]iperf -c 10.10.0.12[/CMD]
------------------------------------------------------------
Client connecting to 10.10.0.12, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 10.10.0.10 port 63461 connected with 10.10.0.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   952 Mbits/sec
Code:
[B][U]OVER NFS[/U][/B]
sync)   68528896 bytes/sec (65 MB/s)
async)  69435545 bytes/sec (66 MB/s)
Tests over NFS have been made like:
Code:
[CMD="#"]dd if=/dev/random of=rndfile bs=1m count=2048[/CMD]
[CMD="#"]dd if=rndfile of=/dev/zero bs=1m[/CMD]
1766810021 (1684 MB/s)

[B]sync)[/B]
[CMD="#"]mount 10.10.0.12:/export/perftest /mnt/perftest[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]umount /mnt/perftest[/CMD]
[B]async)[/B]
[CMD="#"]mount -o async 10.10.0.12:/export/perftest /mnt/perftest[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]dd if=rndfile of=/mnt/perftest/rndfile bs=1m[/CMD]
[CMD="#"]umount /mnt/perftest[/CMD]
So, still what a guest VM can expect to get is about 60MB/s. Highscore´s been updated.

/Sebulon
 

usdmatt

Daemon

Reaction score: 521
Messages: 1,399

Has anyone ever tried playing with the settings (disclaimer: less 'settings' than 'fixed source code constants') mentioned in these messages?

http://lists.freebsd.org/pipermail/freebsd-fs/2012-March/013994.html
http://lists.freebsd.org/pipermail/freebsd-fs/2013-June/017519.html

I assume there's a good reason these are set the way they are, but these users appeared to get fairly substantial performance gains and it looks like Solaris/ESXi/Linux use bigger defaults (or at least support mounting NFS exports with larger values).
 

RusDyr

Member

Reaction score: 15
Messages: 90

65 MBytes/sec over NFS is still too low for that configuration.

And it's much better to test with iozone or sysbench instead of dd - they will automatically test with different sync/async write/read/rewrite.
 
Top