gmirror performance.

[fxp]

Just a single disk:
diskinfo -ctv ad0
Transfer rates:
outside: 102400 kbytes in 0.928828 sec = 110246 kbytes/sec
middle: 102400 kbytes in 1.063225 sec = 96311 kbytes/sec
inside: 102400 kbytes in 1.867590 sec = 54830 kbytes/sec


Mirror if 3 same disks:
diskinfo -ctv /dev/mirror/gm0
Transfer rates:
outside: 102400 kbytes in 1.703981 sec = 60095 kbytes/sec
middle: 102400 kbytes in 1.799062 sec = 56919 kbytes/sec
inside: 102400 kbytes in 2.149057 sec = 47649 kbytes/sec


gstat:
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 142 142 18038 2.1 0 0 0.0 29.7| ad0
0 0 0 0 0.0 0 0 0.0 0.0| DEV/ad0/ad0
0 0 0 0 0.0 0 0 0.0 0.0| acd0
0 0 0 0 0.0 0 0 0.0 0.0| MIRROR/gm0/ad0
0 425 425 54225 2.1 0 0 0.0 89.1| mirror/gm0
0 0 0 0 0.0 0 0 0.0 0.0| DEV/acd0/acd0
0 0 0 0 0.0 0 0 0.0 0.0| MBR/ad0/ad0
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1
0 0 0 0 0.0 0 0 0.0 0.0| DEV/mirror/gm0/mirror/gm0
0 0 0 0 0.0 0 0 0.0 0.0| VFS/ffs.mirror/gm0s1a/mirror/gm0s1a
0 0 0 0 0.0 0 0 0.0 0.0| SWAP/swap/mirror/gm0s1b
0 0 0 0 0.0 0 0 0.0 0.0| MBR/mirror/gm0/mirror/gm0
0 425 425 54225 2.1 0 0 0.0 89.8| mirror/gm0s1
0 142 142 18038 2.1 0 0 0.0 29.2| ad1
0 0 0 0 0.0 0 0 0.0 0.0| DEV/ad0s1/ad0s1
0 0 0 0 0.0 0 0 0.0 0.0| VFS/ffs.mirror/gm0s1d/mirror/gm0s1d
0 0 0 0 0.0 0 0 0.0 0.0| MIRROR/gm0/ad1
0 142 142 18148 2.0 0 0 0.0 28.8| ad2
0 0 0 0 0.0 0 0 0.0 0.0| DEV/mirror/gm0s1/mirror/gm0s1
0 0 0 0 0.0 0 0 0.0 0.0| BSD/mirror/gm0s1/mirror/gm0s1
0 425 425 54225 2.1 0 0 0.0 90.8| mirror/gm0s1a

So, each disk used only for 30% and in the end we got 90%. This just isn`t right :\

gmirror list:
Geom name: gm0
State: COMPLETE
Components: 3
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 1642310077
Providers:
1. Name: mirror/gm0
Mediasize: 250059349504 (233G)
Sectorsize: 512
Mode: r3w3e4
Consumers:
1. Name: ad0
Mediasize: 250059350016 (233G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 938101692
2. Name: ad1
Mediasize: 250059350016 (233G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 3435672881
3. Name: ad2
Mediasize: 250059350016 (233G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 3884381126

[SaveTheRbtz]

did you tried something other than round-robin?

I've read something bad about load, but also there is patch for it http://www.freebsd.org/cgi/query-pr.cgi?pr=113885 you can try it.

And as i know round-robin is good for big files, i don't know how it works with lots of small files.

PS. almost forgot, there was some patch for sendfile to better cache handling and tunable readahead http://www.lexa.ru/nginx-ru/msg19497.html