low perfomance throuch mdconfig layer

[Ole]

Hello,

I've see big difference between "native" ufs partition and ufs-image when working with it through mdconfig. In original, i try setup some "image" for jail, which located in 1 file and which
easy for migrate (transfer) to other station. But low I/O level stopping me.

test on i386 machine (i've used glabel):

df -h /usr
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/usr 44G 13G 27G 33% /usr

Make empty dir for makefs last arguments.

[root@ ~]# mkdir /tmp/empty

Make 10 gb empty image file

[root@ ~]# makefs -s10g /usr/test.img /tmp/empty
Calculated size of `/usr/test.img': 10737418240 bytes, 2 inodes




Extent size set to 8192
density reduced from 1073741825 to 906241
/usr/test.img: 10240.0MB (20971520 sectors) block size 8192, fragment size 1024
using 186 cylinder groups of 55.31MB, 7080 blks, 64 inodes.
super-block backups (for fsck -b #) at:
32, 113312, 226592, 339872, 453152, 566432, 679712,
792992, 906272, 1019552, 1132832, 1246112, 1359392, 1472672,
1585952, 1699232, 1812512, 1925792, 2039072, 2152352, 2265632,
2378912, 2492192, 2605472, 2718752, 2832032, 2945312, 3058592,
3171872, 3285152, 3398432, 3511712, 3624992, 3738272, 3851552,
3964832, 4078112, 4191392, 4304672, 4417952, 4531232, 4644512,
4757792, 4871072, 4984352, 5097632, 5210912, 5324192, 5437472,
5550752, 5664032, 5777312, 5890592, 6003872, 6117152, 6230432,
6343712, 6456992, 6570272, 6683552, 6796832, 6910112, 7023392,
7136672, 7249952, 7363232, 7476512, 7589792, 7703072, 7816352,
7929632, 8042912, 8156192, 8269472, 8382752, 8496032, 8609312,
8722592, 8835872, 8949152, 9062432, 9175712, 9288992, 9402272,
9515552, 9628832, 9742112, 9855392, 9968672, 10081952, 10195232,
10308512, 10421792, 10535072, 10648352, 10761632, 10874912, 10988192,
11101472, 11214752, 11328032, 11441312, 11554592, 11667872, 11781152,
11894432, 12007712, 12120992, 12234272, 12347552, 12460832, 12574112,
12687392, 12800672, 12913952, 13027232, 13140512, 13253792, 13367072,
13480352, 13593632, 13706912, 13820192, 13933472, 14046752, 14160032,
14273312, 14386592, 14499872, 14613152, 14726432, 14839712, 14952992,
15066272, 15179552, 15292832, 15406112, 15519392, 15632672, 15745952,
15859232, 15972512, 16085792, 16199072, 16312352, 16425632, 16538912,
16652192, 16765472, 16878752, 16992032, 17105312, 17218592, 17331872,
17445152, 17558432, 17671712, 17784992, 17898272, 18011552, 18124832,
18238112, 18351392, 18464672, 18577952, 18691232, 18804512, 18917792,
19031072, 19144352, 19257632, 19370912, 19484192, 19597472, 19710752,
19824032, 19937312, 20050592, 20163872, 20277152, 20390432, 20503712,
20616992, 20730272, 20843552, 20956832,
Populating `/usr/test.img'
Image `/usr/test.img' complete

attach loop mount with mdconfig

[root@ ~]# mdconfig -a -t vnode -f /usr/test.img
md0

Benchmark for native partition

[root@ ~]# diskinfo -t ufs/usr
ufs/usr
512 # sectorsize
48657399808 # mediasize in bytes (45G)
95033984 # mediasize in sectors
94279 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
ad:S00UJ10Y223767s0 # Disk ident.

Seek times:
Full stroke: 250 iter in 4.052656 sec = 16.211 msec
Half stroke: 250 iter in 5.019666 sec = 20.079 msec
Quarter stroke: 500 iter in 8.833443 sec = 17.667 msec
Short forward: 400 iter in 2.514402 sec = 6.286 msec
Short backward: 400 iter in 2.170288 sec = 5.426 msec
Seq outer: 2048 iter in 0.666733 sec = 0.326 msec
Seq inner: 2048 iter in 0.645770 sec = 0.315 msec
Transfer rates:
outside: 102400 kbytes in 2.351492 sec = 43547 kbytes/sec
middle: 102400 kbytes in 2.260215 sec = 45305 kbytes/sec
inside: 102400 kbytes in 2.315799 sec = 44218 kbytes/sec

Bench for image

[root@ ~]# diskinfo -t md0
md0
512 # sectorsize
10737418240 # mediasize in bytes (10G)
20971520 # mediasize in sectors

Seek times:
Full stroke: 250 iter in 1.713979 sec = 6.856 msec
Half stroke: 250 iter in 4.588597 sec = 18.354 msec
Quarter stroke: 500 iter in 11.919983 sec = 23.840 msec
Short forward: 400 iter in 8.156246 sec = 20.391 msec
Short backward: 400 iter in 3.161339 sec = 7.903 msec
Seq outer: 2048 iter in 1.270436 sec = 0.620 msec
Seq inner: 2048 iter in 1.350746 sec = 0.660 msec
Transfer rates:
outside: 102400 kbytes in 6.716347 sec = 15246 kbytes/sec
middle: 102400 kbytes in 5.291922 sec = 19350 kbytes/sec
inside: 102400 kbytes in 4.456694 sec = 22977 kbytes/sec

And benchmark of image on amd64:

diskinfo -t md0
md0
512 # sectorsize
10737418240 # mediasize in bytes (10G)
20971520 # mediasize in sectors

Seek times:
Full stroke: 250 iter in 4.998444 sec = 19.994 msec
Half stroke: 250 iter in 5.307308 sec = 21.229 msec
Quarter stroke: 500 iter in 9.208870 sec = 18.418 msec
Short forward: 400 iter in 4.030579 sec = 10.076 msec
Short backward: 400 iter in 4.450998 sec = 11.127 msec
Seq outer: 2048 iter in 15.509073 sec = 7.573 msec
Seq inner: 2048 iter in 13.211030 sec = 6.451 msec
Transfer rates:
outside: 102400 kbytes in 15.928215 sec = 6429 kbytes/sec
middle: 102400 kbytes in 13.462810 sec = 7606 kbytes/sec
inside: 102400 kbytes in 25.917851 sec = 3951 kbytes/sec

When "native" result on some machine:

diskinfo -t aacd0s1f
aacd0s1f
512 # sectorsize
136493421568 # mediasize in bytes (127G)
266588714 # mediasize in sectors
16594 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.

Seek times:
Full stroke: 250 iter in 2.381814 sec = 9.527 msec
Half stroke: 250 iter in 1.723285 sec = 6.893 msec
Quarter stroke: 500 iter in 2.868653 sec = 5.737 msec
Short forward: 400 iter in 1.297411 sec = 3.244 msec
Short backward: 400 iter in 1.417104 sec = 3.543 msec
Seq outer: 2048 iter in 0.900964 sec = 0.440 msec
Seq inner: 2048 iter in 0.892073 sec = 0.436 msec
Transfer rates:
outside: 102400 kbytes in 1.193608 sec = 85790 kbytes/sec
middle: 102400 kbytes in 1.166153 sec = 87810 kbytes/sec
inside: 102400 kbytes in 2.081808 sec = 49188 kbytes/sec

++UPDATE. Low perfomance i see when md0 is mounted. I assume that
difference for 10-20.000 kbytes/sec is the cost in re-presentation mapping mechanism for mdconfig. But when md0 is mounted, I/O is very poor (< 10 Kbytes/sec).