Best ZFS SAS controllers

A: Don't use RAID controllers. Use HBAs.
B: In general, I like LSI Logic = Broadcom = Arago. Part numbers are for example 92xx and 93xx.
 
ralphbsz said:
In general, I like LSI Logic = Broadcom = Arago. Part numbers are for example 92xx and 93xx.
Click to expand...
+1 for LSI/Broadcom/Arago/whatever they're called these days.

Code: 
# mpsutil show adapter
mps0 Adapter:
       Board Name: SAS9207-8i
   Board Assembly: H3-25412-00K
        Chip Name: LSISAS2308
    Chip Revision: ALL
    BIOS Revision: 7.39.00.00
Firmware Revision: 20.00.02.00
  Integrated RAID: no

PhyNum  CtlrHandle  DevHandle  Disabled  Speed   Min    Max    Device
0       0001        0009       N         6.0     1.5    6.0    SAS Initiator
1       0002        000a       N         6.0     1.5    6.0    SAS Initiator
2       0003        000b       N         6.0     1.5    6.0    SAS Initiator
3       0004        000c       N         6.0     1.5    6.0    SAS Initiator
4       0005        000d       N         6.0     1.5    6.0    SAS Initiator
5       0006        000e       N         6.0     1.5    6.0    SAS Initiator
6       0007        000f       N         6.0     1.5    6.0    SAS Initiator
7                              N                 1.5    6.0    SAS Initiator

In general the various mfi(4), mps(4), mpt(4) and mrsas(4) cards all work like a charm.
 
Just noticed that I had a typo in my post above: It's Avago, not Arago. The cards are marketed under the Broadcom name today; they used to be called LSI or LSI Logic.
 
My only caution with the LSI/Avago/Broadcom cards is there are so many versions.
I recently had troubles with SAS9440 tri-mode card bought off ebay as Lenovo version.
So I wondered if I got someones bad flash card as it didn't seem to work.
Liked the card so i purchased a brand new 'Intel Storage Controller' that is their version of the LSI SAS9440
Found out that the Intel cards use a different VID/PID and are not supported on FreeBSD.
I could maybe hack these pcidevs in but I don't care to. Intel does not support FreeBSD with drivers.

So I recommend you stick with a Retail LSI/Avago/Broadcom card. I would stay away from the SAS94xx cards.
Newegg or similar. Not ebay.

There is a whole cottage industry around flashing LSI cards firmware. Buy a cheap OEM LSI and flash it to "IT" mode.
I have done lots of it in the past but the tools for flashing keep changing. The latest require EFI magic.

I recommend Retail SAS9305-16i for 16 internal ports.
 
For 8 Port SAS card from LSI it becomes tougher. The official model is SAS9300-8i.
They do not seem to be for sale anymore from legitimate sellers.
Even my favorite Newegg is using a third party for this item as is Amazon. BEWARE.
https://www.amazon.com/LSI-Broadcom-9300-8i-PCI-Express-Profile/dp/B00DSURZYS
https://www.newegg.com/lsi-9300-8i-sata-sas/p/N82E16816118217

Personally I wrongly spent enough money to say beware of these third party vendors.
For some reason Newegg is still selling/stocking the 16 port version. It looks legit.
The SAS94xx is firmly entrenched in the retail market. So they must still have old inventory.
Remember that the SAS93xx debuted in 2014.
The SAS92xx is getting pretty tired but if you have SAS2 or SATA3 it will suffice. For 80 bucks that is the low limit.
 
I would think that home users and amateurs don't need anything faster than 6GBit SAS or SATA, and PCIe 2.0. If you are connecting hundreds of disk drives, then the fastest 93xx and 94xx drives may be necessary, but few people who discuss with us here on the forum do that kind of things (perhaps with the exception of Terry K).

But you may very well be right, it may be hard to buy the older and cost-effective 92xx and 93xx cards these days. And do beware of used "branded" versions (like Lenovo, Dell, HP, IBM, Oracle ...): as you said, it is sometimes hard or impossible to get firmware updates for these, unless you are a legitimate customer of those companies with a service contract.
 
Comes down to cables too. Would you want to invest in the old cable system or new. SFF-8643 or SFF-8087.
Good cables are expensive.
 
As long as we are talking internal drives, cabling and power distribution is not a big factor. Even with quality cables and quality power supplies, that only adds a few dozen $ to the cost of the disk drive. Given that a good disk drive costs several hundred $, the extra expense is not make-or-break.

Where it gets pricey is when one has enough disk drives to require external disk enclosures, with the attendant power distribution (external JBODs typically have dual power supplies, so it makes sense to connect them to independent power distribution systems), and SAS expanders in the JBODs to get many dozen disk drives funneled into fewer SAS lanes. Plus, with this many drives one typically runs redundant SAS cables (for example two cables to two different HBAs in the same node, for multi-path, or to two different nodes, for failover or parallel access). Still, if an external enclosure can handle ~100 disk drives, spending many thousands of $ on the enclosure and cabling is not a huge factor, compared to the drives themselves.
 
I am talking about Fan-Out cables. SFF-8643 to 4 SATA3 drives (Or 29 pin SAS version).
You can buy a crappy one for $20 but a good Amphenol or Supermicro cable is $50. I shun the cheap ones.
Times 2 channels or Times 4 channels could be up to $200 in cables if doing >12 drives.
It is definitely cheaper to go SAS2/SFF-8087 but SAS3 controllers are the future.
Does SAS4 use the same connectors? That would be a cincher. It is on the horizon.
 
Has anyone had any success with the Intel SCU ports built on lots of Supermicro server motherboards?
These are SAS2/3 connectors tied to a CPU controller/ storage controller-less.
They preform pretty bad from a cursory check.
Intel® RSTe SCU RAID Legacy Option ROM
 
Phishfry said:
Times 2 channels or Times 4 channels could be up to $200 in cables if doing >12 drives.
Click to expand...
Sure, but the 12 disk drives will cost you about $4000, so $200 to hook them up reliably doesn't seem out of line.
[/quote]

Does SAS4 use the same connectors? That would be a cincher. It is on the horizon,.
Click to expand...
Yes and no. At the drive level, SAS 4.0 connectors are compatible with the existing drive connectors, but they also have extra contacts. So you can take a SAS 4.0 disk drive and connect it to a SAS 3.0 cable, and it will work at 3.0 speeds. I don't know what the HBA vendors will do though.

Honestly, for individual spinning disk drives (which have a maximum throughput of ~200 MByte/s each), it doesn't make a difference. Where SAS 4.0 comes in: If you have a limited number of PCIe lanes, and you want to connect hundreds of drives in external enclosures.

Phishfry said:
Has anyone had any success with the Intel SCU ports built on lots of Supermicro server motherboards?
Click to expand...
Never tried one. Supermicro has had a series of espionage / data breach scandals (whether real or staged is an interesting question, and not all of them have been public knowledge), and for that reason I haven't used any Supermicro boards in about 5 years. Which is sad, because they used to make really good hardware that was a joy to use.
 
Phishfry said:
My only caution with the LSI/Avago/Broadcom cards is there are so many versions.
I recently had troubles with SAS9440 tri-mode card bought off ebay as Lenovo version.
So I wondered if I got someones bad flash card as it didn't seem to work.
Liked the card so i purchased a brand new 'Intel Storage Controller' that is their version of the LSI SAS9440
Found out that the Intel cards use a different VID/PID and are not supported on FreeBSD.
I could maybe hack these pcidevs in but I don't care to. Intel does not support FreeBSD with drivers.

So I recommend you stick with a Retail LSI/Avago/Broadcom card. I would stay away from the SAS94xx cards.
Newegg or similar. Not ebay.

There is a whole cottage industry around flashing LSI cards firmware. Buy a cheap OEM LSI and flash it to "IT" mode.
I have done lots of it in the past but the tools for flashing keep changing. The latest require EFI magic.

I recommend Retail SAS9305-16i for 16 internal ports.
Click to expand...
I've done that for myself (2 cards). One was Chinese IBM M5014/M5015 (I assume knockoff) to 'generic' LSI 9660-8i. Difference remained, twice less onboard RAM on card, compared to real LSI card. Otherwise it worked splendidly until I switched to SSD's and sold it. Flashing was required to get around OEM vendor-lock for disks. It only recognized IBM disks until I flashed it. After flashing I even used Dell's (originally fiberoptic Advanced Format I had converted to 512b/s format - FreeBSD ports has sg3_utils if any of you ever needs to do smth like this too) SAS drive in array without problems.
Card worked on FreeBSD just fine but I was limited to using UFS2 in SAS RAID. Adapter's chipset has firmware available for IT mode, so it's unusable for ZFS. Arrays were recognized as PCIe boot drives by motherboard BIOS (3 different, at time where nvme wasnt invented yet)

One Intel SASUC8I. Bought it already flashed to IT mode. Flashed it few times into RAID mode to see what else it could do. No problems on FreeBSD/ZFS if you stuck to IT firmware. Card would pretty much act like plain drive controller, nothing more. I cant remember if it registered in PC as Intel or generic LSI.

Because those cards are server equipment and server cooling is way more aggressive, when you acquire such a card, equip it with custom cooler. Or risk burning your card down.
 
One of the reasons big manufacturers have custom firmware on OEM'ed cards (such as the LSI/Broadcom SAS HBAs, or Mellanox Infiniband) is indeed cooling, or more generally server thermal management. We had a famous incident in our lab, where we were using a pre-release LSI card from vendor X, with a pre-release server also from vendor X. Nobody told us that the BIOS / integrated thermal management of the motherboard didn't know about the card, and forgot to turn the fans on correctly. Oops. The last thing we ever heard from the SAS chip was that it self-reported temperature as 105 degree (C not F). Shortly thereafter we noticed that the server room smelled very bad. The PC board was dark brown around the chip. And this is why for high-reliability systems you are better off buying all components from a single vendor, and making sure that vendor has done "integration" carefully.
 
You must log in or register to reply here.
Back
Top