Well, the card came in today, MUCH sooner than expected (delivery date was April 28th, and here it is ..)
Fitted the card into a Supermicro Chassis to test it, and at first it wasn't even seen (not in BIOS, not in any OS I tried), so that began really well. I then tried another slot, and things started to work. I could boot from it, first thing I tried was Ubuntu 18.04, and that all worked fine. Performance seemed pretty OK as well, on par with what I was hoping for. Next test was FreeBSD of course. Created a USB stick, fitted two 512GB Samsung 840 PRO's, and installed 12.1-RELEASE onto that (ZFS raid0/stripe config). Booted well, no problems.
First, let's see how FreeBSD sees everything:
Code:
# pciconf -lvc
mps0@pci0:4:0:0: class=0x010700 card=0x30c01000 chip=0x00641000 rev=0x02 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]'
class = mass storage
subclass = SAS
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR NS
link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit
cap 11[c0] = MSI-X supports 15 messages, enabled
Table in map 0x14[0x2000], PBA in map 0x14[0x3800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 7 supported
First VF RID Offset 0x0001, VF RID Stride 0x0001
VF Device ID 0x0064
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 000e[190] = ARI 1
/var/run/dmesg.boot entry:
Code:
mps0: <Avago Technologies (LSI) SAS2116> port 0xe000-0xe0ff mem 0xfbb9c000-0xfbb9ffff,0xfbb40000-0xfbb7ffff irq 40 at device 0.0 on pci4
mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
So we can use FreeBSD's mpsutil..
Code:
# mpsutil show adapter
mps0 Adapter:
Board Name: SAS9201-16i
Board Assembly:
Chip Name: LSISAS2116
Chip Revision: ALL
BIOS Revision: 7.37.00.00
Firmware Revision: 19.00.00.00
Integrated RAID: no
PhyNum CtlrHandle DevHandle Disabled Speed Min Max Device
0 N 1.5 6.0 SAS Initiator
1 N 1.5 6.0 SAS Initiator
2 0001 0011 N 6.0 1.5 6.0 SAS Initiator
3 0002 0012 N 6.0 1.5 6.0 SAS Initiator
4 N 1.5 6.0 SAS Initiator
5 N 1.5 6.0 SAS Initiator
6 N 1.5 6.0 SAS Initiator
7 N 1.5 6.0 SAS Initiator
8 N 1.5 6.0 SAS Initiator
9 N 1.5 6.0 SAS Initiator
10 N 1.5 6.0 SAS Initiator
11 N 1.5 6.0 SAS Initiator
12 N 1.5 6.0 SAS Initiator
13 N 1.5 6.0 SAS Initiator
14 N 1.5 6.0 SAS Initiator
15 N 1.5 6.0 SAS Initiator
So far so good!
Code:
# mpsutil show iocfacts
MsgVersion: 02.00
MsgLength: 16
Function: 0x3
HeaderVersion: 34,00
IOCNumber: 0
MsgFlags: 0x0
VP_ID: 0
VF_ID: 0
IOCExceptions: 0
IOCStatus: 0
IOCLogInfo: 0x0
MaxChainDepth: 128
WhoInit: 0x4
NumberOfPorts: 1
MaxMSIxVectors: 0
RequestCredit: 7632
ProductID: 0x2213
IOCCapabilities: 0x1285c <ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
FWVersion: 0x13000000
IOCRequestFrameSize: 32
MaxInitiators: 30
MaxTargets: 756
MaxSasExpanders: 224
MaxEnclosures: 224
ProtocolFlags: 0x3 <ScsiTarget,ScsiInitiator>
HighPriorityCredit: 120
MaxRepDescPostQDepth: 65504
ReplyFrameSize: 32
MaxVolumes: 0
MaxDevHandle: 1026
MaxPersistentEntries: 128
MinDevHandle: 17
Code:
# mpsutil show devices
B____T SAS Address Handle Parent Device Speed Enc Slot Wdt
00 32 4433221102000000 0011 0001 SATA Target 6.0 0001 01 1
00 33 4433221103000000 0012 0002 SATA Target 6.0 0001 00 1
Next up, some performance test. I usually do this with OpenSSL to create fast random data, like so:
openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero > randomfile.bin
While this is writing, I monitor the zpool:
Code:
# zpool iostat 1
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zroot 33.1G 911G 2 676 55.5K 82.9M
zroot 35.1G 909G 0 7.34K 0 919M
zroot 35.1G 909G 0 7.66K 0 980M
zroot 37.0G 907G 0 7.51K 0 923M
zroot 37.0G 907G 0 7.27K 0 930M
zroot 38.9G 905G 0 7.48K 0 935M
zroot 38.9G 905G 0 7.48K 0 957M
zroot 38.9G 905G 0 7.81K 0 983M
zroot 40.9G 903G 0 7.23K 0 903M
zroot 40.9G 903G 0 7.67K 0 982M
zroot 42.8G 901G 0 7.24K 0 903M
zroot 42.8G 901G 0 7.67K 0 981M
zroot 44.7G 899G 0 7.52K 0 921M
zroot 44.7G 899G 0 7.67K 0 982M
zroot 46.7G 897G 0 7.32K 0 911M
zroot 46.7G 897G 0 7.67K 0 982M
zroot 48.6G 895G 0 7.35K 0 899M
zroot 48.6G 895G 0 7.68K 0 983M
zroot 50.5G 893G 0 7.21K 0 901M
zroot 50.5G 893G 0 7.67K 0 981M
zroot 52.5G 892G 0 7.32K 0 897M
zroot 52.5G 892G 0 7.67K 0 982M
zroot 54.4G 890G 0 7.28K 0 910M
zroot 54.4G 890G 0 7.67K 0 982M
zroot 56.3G 888G 0 7.31K 0 913M
zroot 56.3G 888G 0 7.48K 0 957M
zroot 58.3G 886G 25 7.34K 432K 917M
Note that this system has 32GB RAM, so I'm well over that writing data (almost 60GB) and speeds don't really drop so these are pretty real I guess.
Now my CPU isn't that fast (E5-2609 v2) and OpenSSL is close to 100% utilization (yes, AES is enabled in the BIOS), but so are the disks (gstat capture):
Code:
dT: 1.001s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
10 3803 0 0 0.0 3801 462921 2.4 98.8| da0
10 3816 0 0 0.0 3814 464135 2.4 99.7| da1
0 0 0 0 0.0 0 0 0.0 0.0| da0p1
0 0 0 0 0.0 0 0 0.0 0.0| da0p2
10 3803 0 0 0.0 3801 462921 2.4 98.8| da0p3
0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot0
0 0 0 0 0.0 0 0 0.0 0.0| da1p1
0 0 0 0 0.0 0 0 0.0 0.0| da1p2
10 3816 0 0 0.0 3814 464135 2.4 99.7| da1p3
0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot1
Oh, some details about the drives I was using (SMART):
Code:
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 840 PRO Series
Serial Number: Yep
LU WWN Device Id: 5 002538 5a00e5e85
Firmware Version: DXM05B0Q
User Capacity: 512,110,190,592 bytes [512 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Apr 20 17:50:13 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
So.. all in all, I'm not dissapointed! I'm ordering a bunch of SATA power cables now, and hope to test with 16 SSD's connected soon. Will update here!