Some time ago I did some testing with bonnie++ of various RAID types with varying sizes of vdevs. As my test environment was a bit awkward, the measured performance probably doesn't mean much to anybody, but I think it's valid in showing relative performance of the different scenarios.
The disks are 750Gb Seagate SATA (fibre-channel), probably 5-6 years old. The fibre-channel controller is Qlogic 2432 (PCIe)
A few comments on the numbers in general.
The write speed to these drives is obviously rotten. My best guess is it's because they have a custom firmware, and are probably optimized for the specialized storage system I nicked them from.
The disks are connected on a 2Gbit fibre-channel link, so maximum aggregate throughput is limited to 250Mb/sec.
Tests are named Rn-DxV, which is RAID type, number_of_disks, and number_of_vdevs, respectively (r1-10x5 is 5 pairs of mirrors, etc).
To contrast the numbers above I have two other pools on the same server
A raidz of 3 1TB disks (Samsung Spinpoint F3) connected to a cheap PCI card (not PCIe!)
Finally 4 500Gb Samsung Spinpoint F1 connected to onboard sata-ports
The disks are 750Gb Seagate SATA (fibre-channel), probably 5-6 years old. The fibre-channel controller is Qlogic 2432 (PCIe)
A few comments on the numbers in general.
The write speed to these drives is obviously rotten. My best guess is it's because they have a custom firmware, and are probably optimized for the specialized storage system I nicked them from.
The disks are connected on a 2Gbit fibre-channel link, so maximum aggregate throughput is limited to 250Mb/sec.
Tests are named Rn-DxV, which is RAID type, number_of_disks, and number_of_vdevs, respectively (r1-10x5 is 5 pairs of mirrors, etc).
Code:
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
r0-1x1 16384 12452 3.5 12457 0.7 9701 4.1 69571 22.8 76289 5.0 435.6 0.8
r0-3x1 16384 30372 10.5 24044 2.8 22521 3.2 224798 63.4 218721 8.2 490.2 0.6
r0-6x1 16384 61981 21.2 46135 5.4 38096 5.2 241451 68.0 235960 9.9 582.9 0.7
r1-2x1 16384 7827 2.7 6443 0.8 6098 0.9 114324 32.6 133370 5.5 223.3 0.4
r1-4x2 16384 18321 4.5 14529 1.7 13860 2.0 237875 69.2 206736 7.0 434.6 0.6
r1-6x3 16384 30104 7.4 23427 2.7 21700 3.1 237144 74.8 221707 7.6 468.3 0.6
r1-8x4 16384 40132 9.8 31087 3.6 27123 3.9 231847 72.9 232825 8.2 605.7 0.7
r1-10x5 16384 50620 12.3 36697 4.2 33167 4.7 242218 75.4 216236 8.3 520.6 0.7
r1-12x6 16384 58827 20.2 44535 5.5 36431 5.2 248551 70.0 225497 9.4 560.6 0.7
r1-3x1 16384 7811 2.7 6474 0.7 6065 0.9 203852 57.8 181481 7.1 256.4 0.3
r1-4x1 16384 7771 2.7 6406 0.7 6111 0.9 220671 62.7 208167 7.4 239.9 0.3
rz-3x1 16384 16324 4.0 13494 1.6 12118 1.8 147904 43.9 133751 5.6 289.7 0.4
rz-4x1 16384 17522 4.3 13900 1.7 13636 2.1 189491 55.6 194048 9.1 316.7 0.5
rz-5x1 16384 32160 7.9 26957 3.2 24104 3.7 192477 56.3 157096 6.9 328.1 0.6
rz-6x1 16384 35097 8.6 30312 3.6 24239 3.7 172308 50.7 144609 6.7 331.3 0.5
rz-7x1 16384 42069 10.3 35453 4.2 29042 4.4 175760 51.6 180262 7.7 334.5 0.5
rz-8x1 16384 51489 12.6 42490 5.1 32418 4.9 176204 51.8 159905 7.6 337.9 0.6
rz-9x1 16384 64666 16.0 54960 6.6 42246 6.2 191484 56.3 196694 9.3 340.2 0.6
rz-10x1 16384 69135 16.9 59571 7.0 43232 6.3 171677 50.6 180785 8.4 334.8 0.6
rz-10x2 16384 77033 18.7 57714 6.8 46416 6.3 202404 63.2 192582 8.4 383.6 0.6
rz-12x1 16384 89859 22.1 78561 9.1 53677 7.6 182063 52.9 186515 8.3 337.1 0.6
rz-12x4 16384 76116 18.8 66149 7.9 50596 7.0 247209 70.5 227460 9.9 529.4 0.8
rz-10x1 16384 67515 16.6 60519 7.2 42928 6.2 175072 50.7 178989 8.3 337.3 0.6
rz-10x2 16384 69963 17.3 60538 7.4 46981 6.5 169718 49.2 442524 17.6 440.5 0.6
rz-12x1 16384 87009 21.5 78695 9.1 54537 7.8 182325 53.0 186673 8.7 322.3 0.6
rz-12x2 16384 79959 19.7 67420 8.2 49417 7.0 174675 50.7 442809 17.3 452.9 0.6
rz-12x3 16384 61741 15.2 53673 6.6 43119 6.0 206408 59.1 279320 11.9 481.1 0.7
rz-12x4 16384 76765 19.1 66507 8.0 49817 7.0 242528 69.2 230220 10.4 583.1 0.8
rz-10x1 20480 68026 16.7 55075 6.9 42194 6.2 171492 49.8 177911 8.6 211.1 0.5
rz-10x2 20480 69869 17.2 59642 7.3 45595 6.4 190497 55.3 161834 7.6 322.0 0.6
rz-12x1 20480 86353 21.3 76429 9.0 52915 7.6 182706 53.3 187363 8.8 230.5 0.5
rz-12x2 20480 79742 19.6 67190 8.1 48515 6.9 174451 50.6 167175 8.0 296.2 0.5
rz-12x3 20480 61376 15.1 52685 6.9 41649 6.1 177622 51.7 300640 13.0 368.0 0.6
rz-12x4 20480 73108 18.1 66084 8.1 47946 6.7 202205 58.6 286075 12.3 418.5 0.7
r2-4x1 16384 13143 3.2 11001 1.4 10177 1.6 146946 43.5 160208 6.7 328.6 0.5
r2-5x1 16384 16728 4.1 14137 1.7 12561 2.0 191950 57.1 152697 7.3 326.7 0.4
r2-6x1 16384 30136 7.4 25255 3.1 22291 3.5 189985 56.0 179590 8.0 305.9 0.6
r2-7x1 16384 32126 7.9 27515 3.3 23750 3.7 161149 47.7 161066 7.7 339.5 0.5
r2-8x1 16384 42141 10.4 36092 4.3 30090 4.6 159255 47.1 139537 6.7 353.9 0.6
r2-9x1 16384 55639 13.6 42627 5.1 32860 4.8 167834 49.6 165601 7.6 323.4 0.5
r2-10x1 16384 60597 14.9 51777 6.3 42409 6.2 192707 57.1 198280 9.7 335.7 0.6
r2-10x2 16384 36511 12.6 29892 3.6 26526 4.0 212053 61.1 259591 12.0 425.2 0.7A raidz of 3 1TB disks (Samsung Spinpoint F3) connected to a cheap PCI card (not PCIe!)
Code:
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
rz-3x1 16384 20627 5.0 21695 2.6 13700 2.0 95670 28.0 181541 7.3 266.9 0.4
Code:
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
rz-4x1 16384 129757 31.9 119700 13.4 61425 8.3 148144 43.4 136389 6.1 254.6 0.4