FreeBSD 14 dows not see all disks on Broadcom/LSI SAS3408 controller

W

wilschie

I have a Supermicro storage server with a Broadcom/LSI SAS3408 controller (9400-8i). To the controller attached are

- 2 x SATA SSD
- 2 x M.2 NVMe
- 36 x SAS HDD

FreeBSD sees all SAS HDDs and the two M.2 NVMe but only one of the SAS HDDs during installation or at live system with geom list and/or camcontrol devlist. I think settings on the controller are correct because Oracle Solaris and some Linux distributions (Debian, RHEL) can see and use all disks.

Does anyone have an explanation for this?
 
Uhh, you've got something else in the loop you're not mentioning, because the "8i" means there's only 8 internal full speed ports, not 40! What are the nVME devices? How are they directly attached? Are they four or two lane? Also I think you made a typo because you list SATA HDDs but then say SAS HDD twice so it's not clear which one you meant.
 
I am scratching my head too.
Lets assume all 36 SAS are hanging off expander.
Tied to limited ports as seen on some SM expanders (BPN###EL1).
Eight does sound too small but i could see 12-16 (four SAS SFF8643)...
Maybe onboard LSI SAS controller and 9400-8i added....

Perhaps only one M.2 module is showing up on a card meant for two.
Bifurcation in BIOS?
 
I'm sorry for the inaccuracies and missing details. I will ask the seller who assembled the system for more details.

Correction: FreeBSD sees all 36 SAS HDDs and the two M.2 NVMe (480 GB Micron 7450 Pro) but only one of the 2 SATA SSDs (Solidigm D3-S4520).

Seller specification on the invoice says: SAS HDDs and SATA SSDs are connected to: Broadcom HBA 9400-8i, PCIe 3.1 x8 # 05-50008-01, PCIe 3.1 x8, 8 x SAS 12Gb/s or SATA 6Gb/s, 2 x SFF-8643
 
For a real inventory run pciconf -lv. Look for mps0 and other mps*

There is a chance it is only running on two channels (8i). You have to scope out your gear.
 
Really guessing here but there is a chance SATA could be setup in the BIOS as AHCI RAID instead of AHCI.
Check your motherboard Intel RAID settings. If on you may need to load a driver or turn off.
 
480 GB Micron 7450 Pro doesn't exist in form factors other than M.2 for the specified capacity, so those devices are not connected to the 9400-8i unless, as I'm still curious, there's more interposers.

How are the 36 + 2 devices connected to the 9400-8i? Is there an expansion backplane and does it have an identifier?
 
Motherboard: Supermicro H12SSL-NT

Storage controller BIOS topology shows:

Enclosure (Enclosure Level=0) with 24 of the 36 SAS HDDs
Enclosure (Enclosure Level=1) with 12 of the 36 SAS HDDs
Expander 1 with the two SATA SSDs of which FreeBSD only sees one

camcontrol devlist shows one

<ATA INTEL SSDSC2KB48 0120> at scbus target 59 lun 0 (pass26, da25)

geom disk list | grep INTEL shows one

descr: ATA INTEL SSDSC2KB48

And as I said, I can see and use all disks under Oracle Solaris and Debian.
 
Phishfry said:
Really guessing here but there is a chance SATA could be setup in the BIOS as AHCI RAID instead of AHCI.
Check your motherboard Intel RAID settings. If on you may need to load a driver or turn off.
Click to expand...
Searched every entry in the BIOS (so I think), could not find AHCI settings.
 
Thank you all, I have to call it a night, my wife is waiting with dinner :)

The SSDSC2KB48 drives should be the 2 Solidigms. I took the name from the specification of the seller's invoice.
 
You are possibly choking the throughput of your two expanders by the cabling.

If your two SATA drives are attached to anything but internal SATA connectors of motherboard I would change that.

Every lane possible of SAS controller needs to be dedicated to the expanders.
 
Thanks so much for showing the electrical layout of those SuperMicro 36 bay chassis.
I am surprised of the grouping. 24 drives on one expander and 12 on the other.
 
What is this piece of hardware that has 36 devices in an 24/12 configuration plus some extra SATA ports? I guess the 36 SAS things are on a backplane, but they're fine. What's the backplane the SATA things are on if they're being listed as being on an Expander?
 
Yea I was wondering that too. Maybe two 24 port expanders? One only occupied by 12 SAS.
I would really expect 3 backplanes (12 on each one) into one expander.
36 port expander is common thing. I have a few. Astec, ChenBro, Areca

So two expanders surprised me. Dual Port expanders are OK. Rest too slow for my taste.
 
wilschie said:
Expander 1 with the two SATA SSDs of which FreeBSD only sees one
Click to expand...
It's been about 8 years since I last touched a LSI card. I remember that their BIOS allowed one to see all the attached drives, one by one. Make sure you see BOTH of the SATA SSDs, and make sure they have sensible model and serial numbers (check both serial and WWN). For example, make sure the serial numbers of the two drives are different from each other, and match the serial/WWN on their paper label. Many years ago we had trouble with disks that had "no" serial number (actually, their serial number was all zeroes), which lead to fun with multiple disks that had the same serial number, and at different layers of the software stack the number of disks varied.

Just on a lark: Try separating the two SATA devices onto the two expanders. Right now you seem to have them on the same one.

Why do I talk about expanders? Because in my experience (which is very old), expander firmware is even more buggy than HBA firmware. It could be that having two nearly-identical SATA devices on a single expander tickles a bug. Also check with the upstream vendors (SuperMicro) whether the expander firmware could/should be upgraded.
 
H12SSL-NT isn't specced by Supermicro for this chassis so I guess somebody assembled it from parts since the default configs don't do AMD.

Examining the parts list for that chassis gives the manuals for the backplanes. The use of the expander seems to indicate the rear 2.5s are attached to BPN-SAS3-826EL1-N4 possibly using a SFF-8613 fanout on SASP3.

I guess the question/problem is what's the deal with 9400-8i's expander support? Probably a simple fix, for the knowledgeable, since the first drive is available.
 
This is how a Debian system sees the disks with udiskstl status

MODEL REVISION SERIAL DEVICE
INTEL SSDSC2KB480GZ 7CV10120 BTYI2353050F480BGN sdak
INTEL SSDSC2KB480GZ 7CV10120 BTYI23530114480BGN sdal
 
You must log in or register to reply here.
Back
Top