Hi Storage. My adventures with Dell PowerEdge R730, PERC 730 SAS-3 controller (aka LSI aka Avago aka Broadcom SAS3108) and SAS-3 SSD drives continues. In our last episode we switched driver from mfi(4)() to mrsas(4)() since the former was causing kernel panic and the abovementioned controller requires the latter anyway IIUC. Sadly, this led me to some rather puzzling and unpleasant discoveries. So, here goes.
By way of setting the stage and giving you enough context:
- PERC 730 above is SAS-3 capable of 12Gbps transfers, based on LSI SAS 2108 12Gbps ROC,
- SSDs connected to the backplane are these HGST also 12Gbps SAS
Now that we've switched over to `mrsas` driver and managed to install FreeBSD and boot off of one of them, time to look ad `dmesg`:
Ok, I mean its probably not wrong, but I've no clue why it says Avago Invader there. FWIW the very first Google hit leads to Broadcom SAS3108 which never actually mentions Invader anything but I guess its correctly identified:
I also expose enclosures via BIOS so we should see some `ses` items and hopefully `da` drives:
wait ... wat
Why is it measly 150MB/s? And does this line
What is going on here?
Eh, maybe its just talk and real perf will be up to standard?
Lets run hopefully realistic test with fio:
that's not encouraging, right?
Lets straight up write:
I must mention that PERC controller BIOS reports "negotiated speed" with those drives as 6Gbps. That in itself is also a mystery to me. I've no idea why this is so, but even then, shouldn't we be faster? I swear I was getting better perf with this same drive on Dell R720 - previous generation server with PERC 710 re-flashed to work as HBA - and that controller was SAS-2 capped at 6Gbps.
I am a bit lost here. Have my expectations missed the mark yet again?
Is there some voodoo magic device hinting I must do to fix that link speed and gain perf back?
In the immortal words of Dr Seuss "well, what would you do if your mother ... wait ... that happened to you"?
EDIT: for anyone with trust issues (aka me)
By way of setting the stage and giving you enough context:
- PERC 730 above is SAS-3 capable of 12Gbps transfers, based on LSI SAS 2108 12Gbps ROC,
- SSDs connected to the backplane are these HGST also 12Gbps SAS
Now that we've switched over to `mrsas` driver and managed to install FreeBSD and boot off of one of them, time to look ad `dmesg`:
Ok, I mean its probably not wrong, but I've no clue why it says Avago Invader there. FWIW the very first Google hit leads to Broadcom SAS3108 which never actually mentions Invader anything but I guess its correctly identified:
Code:
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 07.709.04.00-fbsd
mrsas0: <AVAGO Invader SAS Controller> port 0x2000-0x20ff mem 0x96200000-0x9620ffff,0x96100000-0x961fffff irq 26 at device 0.0 numa-domain 0 on pci3
mrsas0: FW now in Ready state
mrsas0: Using MSI-X with 72 number of vectors
mrsas0: FW supports <96> MSIX vector,Online CPU 72 Current MSIX <72>
mrsas0: max sge: 0x46, max chain frame size: 0x400, max fw cmd: 0x39f
mrsas0: Issuing IOC INIT command to FW.
mrsas0: IOC INIT response received from FW.I also expose enclosures via BIOS so we should see some `ses` items and hopefully `da` drives:
Code:
Root mount waiting for: CAM usbus0 usbus1 usbus2
ses0 at mrsas0 bus 1 scbus1 target 32 lun 0
ses0: <DP BP13G+EXP 3.35> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number 500056b3f4cbeefe
ses0: 150.000MB/s transfers
ses0: SES Device
ses1 at ahciem0 bus 0 scbus6 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses2 at ahciem1 bus 0 scbus13 target 0 lun 0
ses2: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses2: SEMB SES Device
da0 at mrsas0 bus 1 scbus1 target 0 lun 0
da0: <HITACHI HUSMM141CLAR1600 C342> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 0SYBE15A
da0: 150.000MB/s transfers
da0: 1526185MB (3125627568 512 byte sectors)
ses0: pass0,da0 in 'Drive Slot 00', SAS Slot: 2 phys at slot 0
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500056b3f4cbeeff addr 5000cca050bdde25
ses0: phy 1: SAS device type 0 phy 0
ses0: phy 1: parent 0 addr 0wait ... wat
Why is it measly 150MB/s? And does this line
ses0: phy 0: SAS device type 1 claim its SAS-1 (ses0 is our backplane)? That transfer rate there also suggests that it ids the link as SAS-1 1Gbps.What is going on here?
Eh, maybe its just talk and real perf will be up to standard?
Lets run hopefully realistic test with fio:
Code:
$ fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
...
Run status group 0 (all jobs):
WRITE: bw=157MiB/s (165MB/s), 157MiB/s-157MiB/s (165MB/s-165MB/s), io=9443MiB (9902MB), run=60187-60187msecLets straight up write:
Code:
$ touch test.file
$ gshred -n 1 -s 10G test.file
# don't worry, source and dest are on separate drives here each HGST SAS-3
$ rsync -v -W --info=progress2 test.file da1/
test.file
10,737,418,240 100% 340.98MB/s 0:00:30 (xfr#1, to-chk=0/1)
sent 10,740,039,762 bytes received 35 bytes 352,132,452.36 bytes/sec
total size is 10,737,418,240 speedup is 1.00I must mention that PERC controller BIOS reports "negotiated speed" with those drives as 6Gbps. That in itself is also a mystery to me. I've no idea why this is so, but even then, shouldn't we be faster? I swear I was getting better perf with this same drive on Dell R720 - previous generation server with PERC 710 re-flashed to work as HBA - and that controller was SAS-2 capped at 6Gbps.
I am a bit lost here. Have my expectations missed the mark yet again?
Is there some voodoo magic device hinting I must do to fix that link speed and gain perf back?
In the immortal words of Dr Seuss "well, what would you do if your mother ... wait ... that happened to you"?
EDIT: for anyone with trust issues (aka me)
Code:
$ sudo smartctl -a /dev/da1
... ommitted ...
Transport protocol: SAS (SPL-3)
