There seems to be a major performance regression in handling of scrub operations on ZFS pools under some conditions. I have 3 servers with identical hardware and 32TB ZFS pools on each. All are running 8-STABLE. Each system performs normal ZFS reads at 500MB/sec or faster - the only thing that is slow is the scrub operation.
One of the systems, 8-STABLE as of this morning, zpool/zfs 4/15, scrubs at the expected rate, 500MB/sec+:
This system is running 8-stable as of this morning, zpool/zfs versions 5/28:
Note that even while that system is scrubbing at under 50MB/sec, with disks 90% busy, I can still read from the pool at a reasonable rate of 250MB/sec or so:
A third system is running 8-STABLE from June 6th, and scrubs at the same slow rate as the second system above. In order to rule out any kernel mis-tuning that I might have done, I booted from a mfsBSD ISO (8.2-RELEASE + zfs v28) from March, 2011 and saw the same low performance.
The hardware on each box is dual E5520 CPUs, 48GB RAM, 3Ware 9650SE-16ML with 16 WD2003FYYS drives, exported as individual drives. The OS is on a separate controller / drive - the only thing on the 3Ware/WD2003's is the ZFS pool.
One of the systems, 8-STABLE as of this morning, zpool/zfs 4/15, scrubs at the expected rate, 500MB/sec+:
Code:
(0:15) rz1:/data# zpool status
pool: data
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scan: scrub in progress since Mon Jul 18 18:04:14 2011
907G scanned out of 12.4T at 577M/s, 5h47m to go
0 repaired, 7.16% done
(0:16) rz1:/data# gstat -f twd
dT: 1.008s w: 1.000s filter: twd
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
6 626 626 39071 7.4 0 0 0.0 91.5| label/twd0
5 622 622 39301 7.8 0 0 0.0 91.6| label/twd1
7 618 618 39233 8.3 0 0 0.0 92.9| label/twd2
8 606 606 38816 7.6 0 0 0.0 92.6| label/twd3
5 605 605 39160 7.2 0 0 0.0 91.8| label/twd4
5 597 597 39623 7.4 0 0 0.0 88.3| label/twd5
4 600 600 39880 7.8 0 0 0.0 90.3| label/twd6
4 581 581 39529 8.4 0 0 0.0 90.2| label/twd7
8 586 586 38798 7.4 0 0 0.0 88.5| label/twd8
6 609 609 38759 6.7 0 0 0.0 86.9| label/twd9
9 585 585 38548 8.8 0 0 0.0 92.8| label/twd10
10 619 619 39183 7.7 0 0 0.0 91.3| label/twd11
9 602 602 38793 7.8 0 0 0.0 89.8| label/twd12
9 631 631 38898 7.5 0 0 0.0 90.3| label/twd13
6 612 612 39084 7.4 0 0 0.0 88.9| label/twd14
0 0 0 0 0.0 0 0 0.0 0.0| label/twd15
This system is running 8-stable as of this morning, zpool/zfs versions 5/28:
Code:
(0:11) rz2:/data# zpool status
pool: data
state: ONLINE
scan: scrub in progress since Mon Jul 18 11:19:04 2011
2.22T scanned out of 11.8T at 48.5M/s, 35h32m to go
0 repaired, 18.79% done
(0:12) rz2:/data# gstat -f twd
dT: 1.022s w: 1.000s filter: twd
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 97 97 2976 36.8 0 0 0.0 84.0| label/twd0
10 122 122 3803 72.7 0 0 0.0 100.1| label/twd1
0 98 98 3005 34.7 0 0 0.0 82.4| label/twd2
0 114 114 3496 170.5 0 0 0.0 95.7| label/twd3
10 123 123 3793 129.4 0 0 0.0 99.9| label/twd4
0 114 114 3453 50.4 0 0 0.0 85.5| label/twd5
0 115 115 3516 48.5 0 0 0.0 91.2| label/twd6
0 115 115 3483 69.4 0 0 0.0 90.1| label/twd7
0 115 115 3485 59.2 0 0 0.0 91.8| label/twd8
0 115 115 3516 72.4 0 0 0.0 93.9| label/twd9
0 88 88 2700 36.4 0 0 0.0 80.3| label/twd10
0 94 94 2880 71.2 0 0 0.0 84.9| label/twd11
0 91 91 2813 44.3 0 0 0.0 85.6| label/twd12
0 87 87 2687 31.2 0 0 0.0 78.8| label/twd13
0 90 90 2781 32.0 0 0 0.0 80.4| label/twd14
0 0 0 0 0.0 0 0 0.0 0.0| label/twd15
Code:
(0:13) rz2:/data# dd if=test.iso of=/dev/null bs=1m
29902+0 records in
29902+0 records out
31354519552 bytes transferred in 117.035626 secs (267905771 bytes/sec)
The hardware on each box is dual E5520 CPUs, 48GB RAM, 3Ware 9650SE-16ML with 16 WD2003FYYS drives, exported as individual drives. The OS is on a separate controller / drive - the only thing on the 3Ware/WD2003's is the ZFS pool.