LSI 9211 - is write cache enabled or not?

dvl@

dvl@

Developer
As you can see, my drives in one system are getting 456MB/s sequential read, while the other is doing 265MB/s. The slower system is using the LSI 9211. A question raised was: is write cache enabled on the disks attached to the LSI SATA card?

This is a question I've been unable to answer so far.

The card in question is:

Code: 
# pciconf -lv
mps0@pci0:1:0:0:        class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
    class      = mass storage
    subclass   = SAS


# dmesg
mps0: <LSI SAS2008> port 0x8000-0x80ff mem 0xfde3c000-0xfde3ffff,0xfde40000-0xfde7ffff irq 28 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

The suggested method for discover was camcontrol, which will not work on these devices (see previous post on that issue).

Suggestions?
 
The LSI 9211 is an HBA, and provides straight passthrough of the drives connected to it. There are no caching options for it.

It's very very likely that 6 modern onboard SATA controllers are faster than this HBA. For starters, at the OS level there is a lock for the driver that can serialize some operations that the AHCI equipped box can parallelize.
 
jpaetzel@ said:
The LSI 9211 is an HBA, and provides straight passthrough of the drives connected to it. There are no caching options for it.
Click to expand...

So this HBA won't be disabling the write cache on the disks themselves? That was an option-to-explore put forth on my blog.

It's very very likely that 6 modern onboard SATA controllers are faster than this HBA. For starters, at the OS level there is a lock for the driver that can serialize some operations that the AHCI equipped box can parallelize.
Click to expand...

Sounds like I bought old technology. What HBA would you recommend?

Interestingly, even if I create a zpool with just *one* HDD via this LSI SATA card, it still runs at +50% performance. At first, I thought: for just one HDD, serialization won't be an issue; then I realized, no, it can be.

Let me try the same drive model but directly attached to the MB. BRB...
 
When directly attached to the M/B, bonnie++ shows 266MB/s on sequential writes to the HDD.
 
Are the drives partitioned identically? Writes going half speed while reads are only 14% slower suggests misalignment on the slower drive.
 
Let's go slowly here and be sure we're looking at the same time on both machines. Extra eyes are always helpful, thanks.

First box (faster):
Code: 
ada1 at ahcich3 bus 0 scbus3 target 0 lun 0
ada1: <Hitachi HDS723030BLE640 MX6OAAB0> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad10

script for creating test area:

Code: 
$ sudo ./zfs-cluster-build.sh -c config-zfs-cluster-ada1.sh 
ada1
# gpart create -s GPT ada1
ada1 created
# gpart add -a 1m -t freebsd-zfs -l Bay1.ada1 ada1
ada1p1 added
# gnop create -S 4k gpt/Bay1.ada1
# zpool create -f  -m /mnt examplecluster   /dev/gpt/Bay1.ada1.nop
# gpart show ada1
=>        34  5860533101  ada1  GPT  (2.7T)
          34        2014        - free -  (1M)
        2048  5860530176     1  freebsd-zfs  (2.7T)
  5860532224         911        - free -  (455k)

# zdb | grep ashift
            ashift: 12
# zpool status examplecluster
  pool: examplecluster
 state: ONLINE
  scan: none requested
config:

	NAME                 STATE     READ WRITE CKSUM
	examplecluster       ONLINE       0     0     0
	  gpt/Bay1.ada1.nop  ONLINE       0     0     0

errors: No known data errors

bonnie++:

Code: 
[dan@floater:~/bin/benchmarks] $ cd /mnt
[dan@floater:/mnt] $ sudo mkdir dan
[dan@floater:/mnt] $ sudo chown dan:dan dan
[dan@floater:/mnt] $ cd dan
[dan@floater:/mnt/dan] $ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
floater.unixath 32G   190  99 139791  16 67438   8   507  99 168730   6 231.9   1
Latency             42162us    2713ms    3132ms   35013us     285ms    2045ms
Version  1.96       ------Sequential Create------ --------Random Create--------
floater.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             10164us      77us      85us   20406us      25us      52us
1.96,1.96,floater.unixathome.org,1,1364324804,32G,,190,99,139791,16,67438,8,507,99,168730,6,231.9,1,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,42162us,2713ms,3132ms,35013us,285ms,2045ms,10164us,77us,85us,20406us,25us,52us
[dan@floater:/mnt/dan] $

On the slower box:

Code: 
ada1: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad14

Setting up:

Code: 
$ sudo  ./zfs-cluster-build.sh -c config-zfs-cluster-er-ada1.sh 
Password:
ada1
# gpart create -s GPT ada1
ada1 created
# gpart add -a 1m -t freebsd-zfs -l Bay1.ada1 ada1
ada1p1 added
# gnop create -S 4k gpt/Bay1.ada1
# zpool create -f  -m /mnt examplecluster   /dev/gpt/Bay1.ada1.nop
# gpart show ada1
=>        34  5860533101  ada1  GPT  (2.7T)
          34        2014        - free -  (1M)
        2048  5860530176     1  freebsd-zfs  (2.7T)
  5860532224         911        - free -  (455k)

# zdb | grep ashift
            ashift: 12
            ashift: 12
# zpool status examplecluster
  pool: examplecluster
 state: ONLINE
  scan: none requested
config:

	NAME                 STATE     READ WRITE CKSUM
	examplecluster       ONLINE       0     0     0
	  gpt/Bay1.ada1.nop  ONLINE       0     0     0

errors: No known data errors

and bonnie++

Code: 
[dan@heckler:/mnt] $ sudo mkdir dan
[dan@heckler:/mnt] $ sudo chown dan:dan dan
[dan@heckler:/mnt] $ cd dan
[dan@heckler:/mnt/dan] $ bonnie++ 
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
heckler.unixath 64G    95  99 127757  37 60306  19   229  97 168745  24 179.1   6
Latency             84738us    2234ms    3802ms     244ms     240ms     235ms
Version  1.97       ------Sequential Create------ --------Random Create--------
heckler.unixathome. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 15637  91 +++++ +++ 14694  96 16091  96 +++++ +++ 15288  98
Latency             20797us     163us    5084us   21622us     128us     196us
1.97,1.97,heckler.unixathome.org,1,1364266985,64G,,95,99,127757,37,60306,19,229,97,168745,24,179.1,6,16,,,,,15637,91,+++++,+++,14694,96,16091,96,+++++,+++,15288,98,84738us,2234ms,3802ms,244ms,240ms,235ms,20797us,163us,5084us,21622us,128us,196us
 
NOTE: one reports as Toshiba, the other as Hitachi. They are both the same model
 
Those numbers look reasonable, given the two drives aren't quite identical. They should be nearly the same if the drives were swapped between computers.
 
Maybe we're looking at different numbers. bonnie++ has terrible output.

Hitachi: 139791K/sec sequential write, 168730K/sec sequential read
Toshiba: 127757K/sec sequential write, 168745K/sec sequential read

So the Toshiba is about 9% slower for writes. AFAIR, I've seen that kind of variance between supposedly identical drives. Read speed is effectively the same between both. Swap the drives between computers to see if the numbers are consistent.
 
You must log in or register to reply here.
Back
Top