scsi disk slow working

rafalj

rafalj

Hi. I install freebsd 8, but now I realized that data transfer from disk to disk is about 20-30MB/s but not ~200MB/s. Maybe I don't set something

This is my dmesg:

Code: 
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz (2666.78-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106a5  Stepping = 5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 17179869184 (16384 MB)
avail memory = 16552878080 (15786 MB)
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
 cpu8 (AP): APIC ID: 16
 cpu9 (AP): APIC ID: 17
 cpu10 (AP): APIC ID: 18
 cpu11 (AP): APIC ID: 19
 cpu12 (AP): APIC ID: 20
 cpu13 (AP): APIC ID: 21
 cpu14 (AP): APIC ID: 22
 cpu15 (AP): APIC ID: 23
ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 20090521 tbfadt-707
ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 20090521 tbfadt-707
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <HP ProLiant> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci3: <ACPI PCI bus> on pcib1
ciss0: <HP Smart Array P410i> port 0x4000-0x40ff mem 0xfbc00000-0xfbffffff,0xfbbf0000-0xfbbf0fff irq 28 at device 0.0 on pci3
ciss0: PERFORMANT Transport
ciss0: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci10: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci7: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci11: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci12: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci13: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci14: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci2: <ACPI PCI bus> on pcib8
bce0: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf8000000-0xf9ffffff irq 31 at device 0.0 on pci2
miibus0: <MII bus> on bce0
brgphy0: <BCM5709C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce0: Ethernet address: 00:26:55:82:9e:e4
bce0: [ITHREAD]
bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (4.6.4); Flags (MSI|MFW); MFW (NCSI 1.0.3)
bce1: <HP NC382i DP Multifunction Gigabit Server Adapter (C0)> mem 0xf6000000-0xf7ffffff irq 39 at device 0.1 on pci2
miibus1: <MII bus> on bce1
brgphy1: <BCM5709C 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce1: Ethernet address: 00:26:55:82:9e:e6
bce1: [ITHREAD]
bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (4.6.4); Flags (MSI|MFW); MFW (NCSI 1.0.3)
pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci4: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci15: <ACPI PCI bus> on pcib10
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x1000-0x101f irq 20 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x003b
usbus0: <UHCI (generic) USB controller> on uhci0
uhci1: <UHCI (generic) USB controller> port 0x1020-0x103f irq 23 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x0010
usbus1: <UHCI (generic) USB controller> on uhci1
uhci2: <UHCI (generic) USB controller> port 0x1040-0x105f irq 22 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x0010
usbus2: <UHCI (generic) USB controller> on uhci2
uhci3: <UHCI (generic) USB controller> port 0x1060-0x107f irq 23 at device 29.3 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x0010
usbus3: <UHCI (generic) USB controller> on uhci3
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf5df0000-0xf5df03ff irq 20 at device 29.7 on pci0
 
second part dmesg
Code: 
ehci0: [ITHREAD]
usbus4: waiting for BIOS to give up control
usbus4: EHCI version 1.0
usbus4: <EHCI (generic) USB 2.0 controller> on ehci0
pcib11: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib11
vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xe8000000-0xefffffff,0xf5ff0000-0xf5ffffff irq 23 at device 3.0 on pci1
pci1: <base peripheral> at device 4.0 (no driver attached)
pci1: <base peripheral> at device 4.2 (no driver attached)
uhci4: <UHCI (generic) USB controller> port 0x3800-0x381f irq 22 at device 4.4 on pci1
uhci4: [ITHREAD]
uhci4: LegSup = 0x0010
usbus5: <UHCI (generic) USB controller> on uhci4
pci1: <serial bus> at device 4.6 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est4 attach returned 6
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est5 attach returned 6
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est6 attach returned 6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
est7: <Enhanced SpeedStep Frequency Control> on cpu7
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est7 attach returned 6
p4tcc7: <CPU Frequency Thermal Control> on cpu7
cpu8: <ACPI CPU> on acpi0
est8: <Enhanced SpeedStep Frequency Control> on cpu8
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est8 attach returned 6
p4tcc8: <CPU Frequency Thermal Control> on cpu8
cpu9: <ACPI CPU> on acpi0
est9: <Enhanced SpeedStep Frequency Control> on cpu9
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est9 attach returned 6
p4tcc9: <CPU Frequency Thermal Control> on cpu9
cpu10: <ACPI CPU> on acpi0
est10: <Enhanced SpeedStep Frequency Control> on cpu10
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est10 attach returned 6
p4tcc10: <CPU Frequency Thermal Control> on cpu10
cpu11: <ACPI CPU> on acpi0
est11: <Enhanced SpeedStep Frequency Control> on cpu11
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est11 attach returned 6
p4tcc11: <CPU Frequency Thermal Control> on cpu11
cpu12: <ACPI CPU> on acpi0
est12: <Enhanced SpeedStep Frequency Control> on cpu12
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est12 attach returned 6
p4tcc12: <CPU Frequency Thermal Control> on cpu12
cpu13: <ACPI CPU> on acpi0
est13: <Enhanced SpeedStep Frequency Control> on cpu13
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est13 attach returned 6
p4tcc13: <CPU Frequency Thermal Control> on cpu13
cpu14: <ACPI CPU> on acpi0
est14: <Enhanced SpeedStep Frequency Control> on cpu14
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est14 attach returned 6
p4tcc14: <CPU Frequency Thermal Control> on cpu14
cpu15: <ACPI CPU> on acpi0
est15: <Enhanced SpeedStep Frequency Control> on cpu15
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 15
device_attach: est15 attach returned 6
p4tcc15: <CPU Frequency Thermal Control> on cpu15
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8-0x2ff irq 3 on isa0
uart1: [FILTER]
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
usbus5: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
ugen5.1: <(0x103c)> at usbus5
uhub5: <(0x103c) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
ugen5.2: <HP> at usbus5
ukbd0: <Virtual Keyboard> on usbus5
kbd2 at ukbd0
ums0: <Virtual Mouse> on usbus5
ums0: 3 buttons and [XY] coordinates ID=0
ugen5.3: <HP> at usbus5
uhub6: <Virtual Hub> on usbus5
uhub4: 8 ports with 8 removable, self powered
uhub6: 7 ports with 7 removable, self powered
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 1  VOLUME OK> Fixed Direct Access SCSI-5 device 
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C)
SMP: AP CPU #1 Launched!


---------
What should I do now? This same problem I got on FreeBSD 6.3 but there is SmartArray P400
Maybe I have to set something in kernel?
Please help
 
I haven't used a Compaq SmartArray in 5+ years, but what kind of disks are they and how are you measuring the the transfer rate.
 
Sorry that I not format correctly my last post.

I dont know what disks there are. I only know that 2 disks are 72GB and third is 147GB.
And disks are configured like RAID 10
 
rafalj said:
I dont know what disks there are. I only know that 2 disks are 72GB and third is 147GB.
And disks are configured like RAID 10
Click to expand...
I'm not familiar with recent Smart Array cards, but the HP web site says that the 410 is a PCIe x8 SAS controller, so it should be relatively fast.

Your dmesg output:

Code: 
ciss0: <HP Smart Array P410i> port 0x4000-0x40ff mem 0xfbc00000-0xfbffffff,0xfbbf0000-0xfbbf0fff irq 28 at device 0.0 on pci3
ciss0: PERFORMANT Transport
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 1  VOLUME OK> Fixed Direct Access SCSI-5 device 
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C)
Click to expand...

says that the controller is providing a single RAID 1 (mirroring) drive. As another user asked, how are you testing the speed? Your original post says "data transfer from disk to disk ". I only see the one disk in your dmesg output - copying data to the same drive you're reading from will be a lot slower than either reading or writing alone, since the disk heads have to move back and forth. It looks like that controller has 256MB of memory, so any writes that are larger than that won't be cached and will have to wait for the drives.
 
Sorry for my mistake. That dmesg from other machine with only one drive. This secound one (with three drives) dmesg don't show me anything before I reset it.
Thats output from mount

3 drives

Code: 
/dev/da0s2a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/da2s1d on /www2 (ufs, local, soft-updates)
/dev/da1s1d on /mysql-da1s1 (ufs, local, soft-updates)
procfs on /proc (procfs, local)
devfs on /var/named/dev (devfs, local)
How I measure? I try to copy one big file from one drive to another. I copy it be Midnight Commander. The speed is ~15MB/s, sometimes comes to 30MB/s. But when I try copy this file ones again speed is ~100MB/s.

When I try to resolve it I read a lot of threats on forums and I see that someone has got speed ~300MB/s on this same controller and drives? In BIOS I set BBU to enable and write-thru to enable?

How you got any ideas?
 
And second measure. Is it normal speed?
Code: 
dd if=/dev/da0 of=/dev/null
^C377875+0 records in
377875+0 records out
193472000 bytes transferred in 27.763154 secs (6968661 bytes/sec)
 
And second server (this is with three disks)
Code: 
dd if=/dev/da0 of=/dev/null
^C3300+0 records in
3300+0 records out
1689600 bytes transferred in 3.527764 secs (478944 bytes/sec)

dd if=/dev/da1 of=/dev/null
^C2790+0 records in
2790+0 records out
1428480 bytes transferred in 2.865040 secs (498590 bytes/sec)

dd if=/dev/da2 of=/dev/null
^C3346+0 records in
3346+0 records out
1713152 bytes transferred in 3.523210 secs (486247 bytes/sec)

Sorry for multipost but I dont know how to edit my posts
 
rafalj said:
What can I do more?
Click to expand...

give us exact informations

about what drives you are using (manufactor/model)

how your raid controller (the arrays) is configured

and please use a benchmark tool like iozone instead of dd
 
ok I install iozone. What data you want to see? Which syntax to use?
Info about array and disk I try to check today
 
Code: 
iozone -R -l 1 -u 1 -r 8k -s 16384m -F /home/f0 >> /tmp/iozone_results.txt

But you should use a filesize equal or bigger than your RAM. I guess 16384m.
Maybe more concurrence processes too.

Code: 
iozone -R -l 4 -u 4 -r 8k -s 16384m -F /home/f0 /home/f1 /home/f2 /home/f3 >> /tmp/iozone_results.txt

while iozone is running you can use

Code: 
systat -vm
or
Code: 
systat -io

to find out what happens atm on your arrays/disks
 
I type it, but I don't know how to read it. For me nothing special.
I also find another measure program, here is summary:

First one:
Code: 
diskinfo -c /dev/da0
/dev/da0
	512         	# sectorsize
	73372631040 	# mediasize in bytes (68G)
	143305920   	# mediasize in sectors
	17562       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

I/O command overhead:
	time to read 10MB block      0.425855 sec	=    0.021 msec/sector
	time to read 20480 sectors  21.308706 sec	=    1.040 msec/sector
	calculated command overhead			=    1.020 msec/sector

[root@czesio /home/rafal]# diskinfo -t /dev/da0
/dev/da0
	512         	# sectorsize
	73372631040 	# mediasize in bytes (68G)
	143305920   	# mediasize in sectors
	17562       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

Seek times:
	Full stroke:	  250 iter in   2.388086 sec =    9.552 msec
	Half stroke:	  250 iter in   2.112366 sec =    8.449 msec
	Quarter stroke:	  500 iter in   3.755740 sec =    7.511 msec
	Short forward:	  400 iter in   1.362590 sec =    3.406 msec
	Short backward:	  400 iter in   1.367906 sec =    3.420 msec
	Seq outer:	 2048 iter in   2.242906 sec =    1.095 msec
	Seq inner:	 2048 iter in   2.300761 sec =    1.123 msec
Transfer rates:
	outside:       102400 kbytes in   1.629397 sec =    62845 kbytes/sec
	middle:        102400 kbytes in   1.456342 sec =    70313 kbytes/sec
	inside:        102400 kbytes in   2.482639 sec =    41246 kbytes/sec

Second one:
Code: 
diskinfo -c /dev/da1
/dev/da1
	512         	# sectorsize
	73372631040 	# mediasize in bytes (68G)
	143305920   	# mediasize in sectors
	17562       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

I/O command overhead:
	time to read 10MB block      0.225791 sec	=    0.011 msec/sector
	time to read 20480 sectors  21.048077 sec	=    1.028 msec/sector
	calculated command overhead			=    1.017 msec/sector

[root@czesio /home/rafal]# diskinfo -t /dev/da1
/dev/da1
	512         	# sectorsize
	73372631040 	# mediasize in bytes (68G)
	143305920   	# mediasize in sectors
	17562       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

Seek times:
	Full stroke:	  250 iter in   0.984033 sec =    3.936 msec
	Half stroke:	  250 iter in   1.029704 sec =    4.119 msec
	Quarter stroke:	  500 iter in   2.095782 sec =    4.192 msec
	Short forward:	  400 iter in   1.580941 sec =    3.952 msec
	Short backward:	  400 iter in   1.736556 sec =    4.341 msec
	Seq outer:	 2048 iter in   2.069760 sec =    1.011 msec
	Seq inner:	 2048 iter in   2.175536 sec =    1.062 msec
Transfer rates:
	outside:       102400 kbytes in   2.138406 sec =    47886 kbytes/sec
	middle:        102400 kbytes in   2.499299 sec =    40971 kbytes/sec
	inside:        102400 kbytes in   3.224869 sec =    31753 kbytes/sec
and third one:
Code: 
 diskinfo -c /dev/da2
/dev/da2
	512         	# sectorsize
	299966445568	# mediasize in bytes (279G)
	585871964   	# mediasize in sectors
	71798       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

I/O command overhead:
	time to read 10MB block      0.133843 sec	=    0.007 msec/sector
	time to read 20480 sectors  21.012034 sec	=    1.026 msec/sector
	calculated command overhead			=    1.019 msec/sector

[root@czesio /home/rafal]# diskinfo -t /dev/da2
/dev/da2
	512         	# sectorsize
	299966445568	# mediasize in bytes (279G)
	585871964   	# mediasize in sectors
	71798       	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	32          	# Sectors according to firmware.

Seek times:
	Full stroke:	  250 iter in   0.966620 sec =    3.866 msec
	Half stroke:	  250 iter in   0.964072 sec =    3.856 msec
	Quarter stroke:	  500 iter in   2.050793 sec =    4.102 msec
	Short forward:	  400 iter in   1.431910 sec =    3.580 msec
	Short backward:	  400 iter in   1.222912 sec =    3.057 msec
	Seq outer:	 2048 iter in   2.134341 sec =    1.042 msec
	Seq inner:	 2048 iter in   2.097480 sec =    1.024 msec
Transfer rates:
	outside:       102400 kbytes in   1.291908 sec =    79263 kbytes/sec
	middle:        102400 kbytes in   1.627644 sec =    62913 kbytes/sec
	inside:        102400 kbytes in   1.523108 sec =    67231 kbytes/sec

Is this speed is normal? My MySQL database hangs up where more people want to read in this same time
 
rafalj said:
Code: 
da0

Transfer rates:
	outside:       102400 kbytes in   1.629397 sec =    62845 kbytes/sec
	middle:        102400 kbytes in   1.456342 sec =    70313 kbytes/sec
	inside:        102400 kbytes in   2.482639 sec =    41246 kbytes/sec

da1

Transfer rates:
	outside:       102400 kbytes in   2.138406 sec =    47886 kbytes/sec
	middle:        102400 kbytes in   2.499299 sec =    40971 kbytes/sec
	inside:        102400 kbytes in   3.224869 sec =    31753 kbytes/sec

da2

Transfer rates:
	outside:       102400 kbytes in   1.291908 sec =    79263 kbytes/sec
	middle:        102400 kbytes in   1.627644 sec =    62913 kbytes/sec
	inside:        102400 kbytes in   1.523108 sec =    67231 kbytes/sec

Is this speed is normal? My MySQL database hangs up where more people want to read in this same time
Click to expand...

Is the server running mysql with active users while you try to benchmark your diskspeed?
 
Yes. In this server MySQL and apache is working while I testing speed
Root is on da0
www is on da1
mysql-root is on da2
 
rafalj:

1) You have not provided us with information as to what specifically those drives are. (i.e. no dmesg output describing speeds for the 3 disks)

2) The numbers listed in dmesg output are maximum theoretical speeds of the hardware. The closest you'll get to these speeds are by way of sequential reads/writes or by using a BBC (Battery Backed Cache) on the array assuming the disks you mentioned are attached to the array.

3) dd by default uses 1 block per iteration. This is slow. Use 4k blocks or larger. Example:

Code: 
nat# dd if=/dev/ad0s1e of=/dev/null bs=4096
131072+0 records in
131072+0 records out
536870912 bytes transferred in 25.211596 secs (21294602 bytes/sec)
nat# dd if=/dev/ad0s1e of=/dev/null 
1048576+0 records in
1048576+0 records out
536870912 bytes transferred in 80.982048 secs (6629505 bytes/sec)

4) Lastly there are performance tunings in regards to block sizes of your file systems. I won't get into details here as there are plenty of references elsewhere, but one important one to keep in mind is: Do not exceed 64k block sizes otherwise the kernel must do a read-modify-write for all operations. With raid setups this is especially true. Read about DFLTPHYS via google for reasons why. I'll get you started:

http://old.nabble.com/ATA-4K-sector-issues-td27929909i20.html

Hope this helps.
 
ad1) As I wrote earlier my every disk is:
Code: 
<COMPAQ RAID 1  VOLUME OK> Fixed Direct Access SCSI-5 device 
135.168MB/s transfers
Command Queueing enabled
139979MB (286677120 512 byte sectors: 255H 32S/T 35132C)
Only diference is their size.

ad3) As you wrote I type dd for 4k block and
Code: 
#dd if=/dev/da0 of=/dev/null bs=4096
4756782+0 records in
4756782+0 records out
19483779072 bytes transferred in 280.838499 secs (69377166 bytes/sec)

My last question is: Should I tuning kernel? Is this help for something?
 
rafalj said:
ad1) As I wrote earlier my every disk is:
Code: 
<COMPAQ RAID 1  VOLUME OK> Fixed Direct Access SCSI-5 device 
135.168MB/s transfers
Command Queueing enabled
139979MB (286677120 512 byte sectors: 255H 32S/T 35132C)
Only diference is their size.

ad3) As you wrote I type dd for 4k block and
Code: 
#dd if=/dev/da0 of=/dev/null bs=4096
4756782+0 records in
4756782+0 records out
19483779072 bytes transferred in 280.838499 secs (69377166 bytes/sec)

My last question is: Should I tuning kernel? Is this help for something?
Click to expand...

So, this is good news. By using 4k Block sizes you've gained roughly 40% throughput on your previous test. You still have more room to go though. Use 8k block sizes and I believe you'll be right under your maximum theoretical throughput.

Code: 
nat# dd if=/dev/ad0s1e of=/dev/null bs=4096
131072+0 records in
131072+0 records out
536870912 bytes transferred in 25.795943 secs (20812223 bytes/sec)
nat# dd if=/dev/ad0s1e of=/dev/null bs=8192
65536+0 records in
65536+0 records out
536870912 bytes transferred in 13.321220 secs (40301933 bytes/sec)

Not too bad for my udma66 connected harddrive. By using 8k blocks per iteration I obtain the highest sequential read performance my old IDE drives are able to do. 16k blocks changes nothing for me. I'm guessing this'll be much the same for you. Go ahead and use 8k blocks, and let me know what your results are. You should (speculation) see nearly double the sequential read performance.

Now a couple of things to keep in mind. 1) If you are doing this on a live server, it's probably best to get a spare up and running for testing. You don't want to piss people off. 2) Mysql, Apache, etc., read/write data all over the file system. They may perform sequential reads and writes, but never to the same file. There is seeking involved which slows down your operations. 3) If you want the absolute best write performance, get a battery backed cache. This allows the controller to tell the OS that the write was completed without it actually being completed and is incredibly fast. This allows the controller to organize its writes and cache pending ones without fear of power failures. 4) Reading the same data over and over again is fast...the OS caches the more recently used blocks AFAIK. 5) Lastly, these are artificial tests. Again, this is because no application that comes to mind performs sequential reads/writes in the manner we are testing.
 
Thanks for your reply.
In fact when I change to 8k block this speeds up
Code: 
# dd if=/dev/da0 of=/dev/null bs=8192
^C369948+0 records in
369948+0 records out
3030614016 bytes transferred in 24.768320 secs (122358481 bytes/sec)
So if I understand all posts my disks works good and there is nothing to do?
 
rafalj said:
Thanks for your reply.
In fact when I change to 8k block this speeds up
Code: 
# dd if=/dev/da0 of=/dev/null bs=8192
^C369948+0 records in
369948+0 records out
3030614016 bytes transferred in 24.768320 secs (122358481 bytes/sec)
So if I understand all posts my disks works good and there is nothing to do?
Click to expand...

Given this latest test, you ARE in fact able to obtain speeds that are close to their maximal theoretical throughput. There is no amount of OS tuning that can speed up these sequential reads/writes.

With that said, (and I've said this already), the real world bottleneck is when you don't perform sequential reads/writes. The iozone mentioned above stresses this to the max. Get a BBC (Battery Backed Cache) and that all changes. If you were to go that route, you'd find even iozone hitting the maximal throughput.

But alas, the BBC costs money and may not be in your budget. If that's the case then you are good to go. Just make sure you've got softupdates on the necessary partitions (assuming ufs) and let her fly.
 
You must log in or register to reply here.
Back
Top