Performance issues over time with ZFS

arad85 · Jun 9, 2012

Hi again,

Further to my previous posts, I am seeing issues over time with zfs performance.

If I leave the system running for 2+ weeks and do the following benchmarks I get:

Code:

[andy@MAINSERVER ~]$ sudo bonnie++ -d /storage/temp/dir -u 0:0  -s 20g
...
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
MAINSERVER      20G   150  99 99443  26 54030  13   340  91 140299  14 105.4   7
Latency             93065us    6100ms    2606ms     630ms     568ms    1210ms
Version  1.96       ------Sequential Create------ --------Random Create--------
MAINSERVER          -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24099  91 +++++ +++ 16864  67 23505  87 +++++ +++ 25727  96
Latency             10047us     116us    8712us   18544us      40us      95us

Rebooting and doing nothing I am seeing:

Code:

[andy@MAINSERVER ~]$  sudo bonnie++ -d /storage/temp/dir -u 0:0  -s 20g
...
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
MAINSERVER      20G   151  99 223546  58 105195  25   366  97 285066  28 104.2   6
Latency               152ms    2482ms    2432ms     121ms     462ms     908ms
Version  1.96       ------Sequential Create------ --------Random Create--------
MAINSERVER          -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23602  91 +++++ +++ 21375  90 20619  85 +++++ +++ 19788  74
Latency             12456us     125us     177us   30806us      77us    2527us

Which is 2x quicker.

I am running 9.0-RELEASE on an AMD Athlon II x4 630 with 10G of memory. The array is a RAIDZ of 4 x Samsung 2TB disks that are aligned with ashift=12.

Any ideas? (I can see a long debug cycle as I wait for this to slow again...).

Performance issues over time with ZFS

arad85