raidz performance tuning question

iwiied · Oct 10, 2012

I am experiencing a performance issue with my current NAS box. When doing a "copying many large files to NAS folder" at the same time with playing a 30gb bluray mkv over a gigabit network, every so often about 4,5 min, I would experience a freeze on the player, which will then recover after about 15 seconds or so.

my system:
freenas 8.02 release 64bit
4x 2tb hitachi 7200k drive configured in a raidz
8gb ddr2
q6600 quad core cpu
onboard built-in gigabit adapter

I've tried to run "zpool iostat 1" to monitor the drive while the operations were going, everything goes smoothly and I can see the 4 drives get written onto every couple seconds. The part when the video freeze is at the end of the paste where the system writes 11 operations in a row instead of 4. Does anyone have any idea what I can do to prevent this from happening? This is my first time using a non-window os. Thank you.

Code:

Raid5       2.86T  4.39T     27    322  3.42M  40.3M
Raid5       2.86T  4.39T      0  2.30K      0   293M
Raid5       2.86T  4.39T      0  2.38K      0   303M
Raid5       2.86T  4.39T     26  1.80K  3.29M   209M
Raid5       2.86T  4.39T     49      3  6.21M  8.77K
Raid5       2.86T  4.39T     13      0  1.69M      0
Raid5       2.86T  4.39T     13      0  1.69M      0
Raid5       2.86T  4.39T     28      0  3.39M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     21      0  2.66M      0
Raid5       2.86T  4.39T     33      0  4.14M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    109      0  13.6M      0
Raid5       2.86T  4.39T      0    722      0  90.4M
Raid5       2.86T  4.39T      0  2.13K      0   272M
Raid5       2.86T  4.39T      0  2.06K      0   263M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      7      0  1002K      0
Raid5       2.86T  4.39T     15      0  1.96M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     27      0  3.41M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      6    735   882K  89.8M
Raid5       2.86T  4.39T      0  2.31K      0   294M
Raid5       2.86T  4.39T     29  1.90K  3.67M   237M
Raid5       2.86T  4.39T     64  1.82K  8.01M   215M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     54      0  6.82M      0
Raid5       2.86T  4.39T      2      0   375K      0
Raid5       2.86T  4.39T     10      0  1.35M      0
Raid5       2.86T  4.39T     14      0  1.83M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     55      0  6.82M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    109      0  13.5M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0    107      0  13.5M
Raid5       2.86T  4.39T      0  2.11K      0   271M
Raid5       2.86T  4.39T      0  2.21K      0   283M
Raid5       2.86T  4.39T      6  1.27K   870K   162M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     55      0  6.88M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    110  1.02K  13.8M   129M
Raid5       2.86T  4.39T      0  2.39K      0   304M
Raid5       2.86T  4.39T      0  2.38K      0   303M
Raid5       2.86T  4.39T     47   1010  5.93M   106M
Raid5       2.86T  4.39T      6      0   866K      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     19      0  2.42M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    109      0  13.5M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      6      0   867K      0
Raid5       2.86T  4.39T      0      3    495   495K
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     22      0  2.78M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0   124K      0
Raid5       2.86T  4.39T     16      0  2.05M      0
Raid5       2.86T  4.39T      0    789      0  98.7M
Raid5       2.86T  4.39T      0  2.33K      0   298M
Raid5       2.86T  4.39T      0  2.01K      0   257M
Raid5       2.86T  4.39T     23  1.42K  2.91M   181M
Raid5       2.86T  4.39T     58      0  7.14M      0
Raid5       2.86T  4.39T     25      0  3.14M      0
Raid5       2.86T  4.39T     19      0  2.41M      0
Raid5       2.86T  4.39T     11      0  1.45M      0
Raid5       2.86T  4.39T      3  1.58K   507K   200M
Raid5       2.86T  4.39T     19  1.71K  2.45M   210M
Raid5       2.86T  4.39T     18  2.28K  2.32M   289M
Raid5       2.86T  4.39T     68  1.19K  8.35M   136M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     55      0  6.79M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    109      0  13.7M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     11      0  1.46M      0
Raid5       2.86T  4.39T      3      0   499K      0
Raid5       2.86T  4.39T      6      0   877K      0
Raid5       2.86T  4.39T      1      0   251K      0
Raid5       2.86T  4.39T      4      0   627K      0
Raid5       2.86T  4.39T     27      0  3.29M      0
Raid5       2.86T  4.39T      1      0   251K      0
Raid5       2.86T  4.39T      7      0  1003K      0
Raid5       2.86T  4.39T     27      0  3.41M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     54      0  6.83M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0    176      0  22.0M
Raid5       2.86T  4.39T      0  2.24K      0   287M
Raid5       2.86T  4.39T      0  2.14K      0   274M
Raid5       2.86T  4.39T      0  2.29K  5.89K   293M
Raid5       2.86T  4.39T    144    769  18.1M  96.2M
Raid5       2.86T  4.39T      0  1.74K      0   221M
Raid5       2.86T  4.39T      0  2.20K      0   280M
Raid5       2.86T  4.39T      1  2.39K   250K   298M
Raid5       2.86T  4.39T     56    477  7.06M  42.8M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     37      0  4.63M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T    109      0  13.7M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      1      0   247K      0
Raid5       2.86T  4.39T     16      0  2.05M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      2      0   371K      0
Raid5       2.86T  4.39T      8      0  1.09M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     23      0  2.79M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T     54      0  6.83M      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0   125K      0
Raid5       2.86T  4.39T     45      0  5.73M      0
Raid5       2.86T  4.39T      0  1.23K   126K   157M
Raid5       2.86T  4.39T      0  1.34K      0   171M
Raid5       2.86T  4.39T      9  2.19K  1.25M   280M
Raid5       2.86T  4.39T      0  1.42K      0   182M
Raid5       2.86T  4.39T      0  1.40K      0   179M
Raid5       2.86T  4.39T      0  1.38K      0   168M
Raid5       2.86T  4.39T      0  1.29K      0   164M
Raid5       2.86T  4.39T      0    951      0   118M
Raid5       2.86T  4.39T      0  1.50K   125K   191M
Raid5       2.86T  4.39T     16  1.79K  2.07M   227M
Raid5       2.86T  4.39T    145  1.21K  18.0M   126M
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0      0      0
Raid5       2.86T  4.39T      0      0  5.83K      0
Raid5       2.86T  4.39T    110      0  13.5M      0

AndyUKG · Oct 10, 2012

Hi,

see this guide, particularly anything that mentions improvements to "bursty" stalls:

http://wiki.freebsd.org/ZFSTuningGuide

Basically you'll need to test some different values and see which gives the most consistent performance under the loads you are testing with...

thanks, Andy.

iwiied · Oct 10, 2012

Thank you Andy, I will give that a try.

Mussolini · Oct 10, 2012

Maybe you have a network issue. Did you monitor the network interface? What's the throughput during the copy/play?

iwiied · Oct 10, 2012

can you tell me the command to do this and what to look for? the only network monitoring i did was the "reporting" web ui provided by freenas. it showed about 200-300 Mb/s rx and 80Mb/s for tx as a rough estimate.
thanks in advance

Mussolini said:
Maybe you have a network issue. Did you monitor the network interface? What's the throughput during the copy/play?

Mussolini · Oct 11, 2012

There are nice tools to do that, like bwm-ng and iftop which can be installed via ports.

Orum · Oct 17, 2012

Are you using samba (smbd) to share? Not sure if what you're experiencing is the same as what I was, as I have a different environment (FreeBSD 9.0 amd64), but I had an issue where prolonged writes at high throughput (80+ MB/s) would intermittently hang, for no apparent reason.

Long story short, I solved it by setting min receivefile size to a nonzero value. In my case, setting

Code:

min receivefile size = 16384

solved my minimum throughput issue.

raidz performance tuning question

iwiied

AndyUKG

iwiied

Mussolini

iwiied

Mussolini

Orum