FreeBSD 14 dows not see all disks on Broadcom/LSI SAS3408 controller

msplsh said:
Honestly, it's probably something dumb like this:

https://forums.freebsd.org/threads/...uns-are-detected-tws-driver.58128/post-332919

Can you figure out which driver FreeBSD is using
Click to expand...

pciconf -lv shows

Sorry for the screenshot. I do not know how to copy and paste from an IPMI session

Screenshot 2024-01-11 at 15.16.39.png
 
Well my suggestion is going to get technical. If too much say so.

What I would do is look at DMESG. When the controller card fires up it has a 'firmware' onboard.
If the FreeBSD driver is too far removed from this firmware weird things happen.
Personally I try and flash the same version firmware on card that FreeBSD uses.
I buy lots of ebay crap so flashing is a first stop for me.
Also I did SAS9400 experiments with TriMode and flashing firmware to NVMe only use.
There are several firmware available for the SAS9400 so I am curious which it is using.
NVMe Only
SAS/SATA Only
Mixed

I think it pays to at least investigate from dmesg|more what version firmware is on card and what version driver is in use. dmesg|grep mpr should get you in the area of concern,

That is stop one on debugging these issues.

Maybe Linux has flashed a newer firmware on card than FreeBSD driver likes. I have seen it happen.
Dual booting LSI is one of the worst things you can do in my experience.
There is alot going on with that firmware/driver interaction.
 
wilschie said:
understand what problem is being described
Click to expand...
Your situation involves expanders on the bus and SAS equipment. This patch seems to acknowledge that there is a problem with mpr involve these things and attempt to correct it. However, the involved parties gave up as they lost access to equipment on which to test the patch and the patch needed more work.

Unless you're willing to let somebody noodle around on your system for an indeterminate amount of time (and they're willing to do it) I think it does mean that you are out of luck.
 
wilschie said:
Does that mean I have to give up?
Click to expand...
Well that approach to enclosure drive assignments was abandoned.
There could be subsequent fixes. Take old PR's for info use only. That was a problem in 2018.
Does it still exist?
Drive mapping tables sounds like it could be similar to your issue. Is it the same?

I can't imagine you are the first to use a common SuperMicro 36bay chassis.

Ohhh yea. What about expander firmware.... There are multiple angles to consider.

But back to square one.
Why exactly are the SATA drives hooked to the expander. What does that bring you?

Commonly a pair of drives like this would hold your OS in a mirror.
I see no problem with motherboard SATA. It allows hotswap.
Why add another protocol to your topography. Expander All SAS. Rear SATA drives use motherboard.
 
My personal summary: Given that Linux can work fine, it must be some odd interaction between the FreeBSD driver stack and all the firmware running below it (HBA, expanders, drives). And that firmware is very complex; I've had to modify expander firmware as part of a previous job, and there are easily 100K lines of code in there. The bug quoted above indicates that this area of device discovery through expanders has had trouble before.

Definitely open a bug report (a PR), just so developers know. Don't expect that to get the problem fixed, in particularly not quickly. This problem can likely not be reproduced without access to your hardware. But it's good for developers to get statistics of what's wrong out there.

For lack of a good idea, I like Phishfry's suggestion of just moving the troubling SATA devices onto motherboard ports. There is however a problem that might prevent that, namely if they have to be hot swappable.
 
Thank you very much for your patient help and all the insights!

I'm not dual booting and I did not flash anything. I also did not assemble and wire the machine myself. The firmware is exactly the same as when delivered, at version 24.00.00.00, which is actually the latest version available from Supermicro.

This is also reported by FreeBSD's dmesg:

Code: 
mpr0: Firmware: 24.00.00.00. Driver: 23.00.00.00-fbsd

A Linux live system dmesg shows

Code: 
mpt3sas_cm0: FW Package Ver(24.00.00.00)

The system is 500 kilometers away from me, so rewiring is not possible.

I think it's pointless to go on here. So Linux it will be. Too bad. It's a lot more fun (for me!) to work with and learn FreeBSD :-) but in the end, the systems have to go into operation at some point. I will try to find out how to create a bug report.
 
wilschie said:
The system is 500 kilometers away from me, so rewiring is not possible.
Click to expand...
Well, there go a lot of good ideas. This also makes some diagnostics impossible. In view of this, and to get the system to production, your decision of using Linux seems very sensible.

The firmware is exactly the same as when delivered, at version 24.00.00.00, which is actually the latest version available from Supermicro.
Click to expand...
That 24.00... is the firmware version of the LSI/Broadcom HBA. But there is a lot more firmware in the box. The one I'm particularly worried about is the firmware of the SAS expanders. Those are the chips on the storage backplane, which take the 8 SAS links that come from the LSI/Broadcom HBA, and split it into several dozen SAS links going to each disk. The backplanes are sold and made by Supermicro. The question here (and it is not a question you as an end user and administrator should be worrying about) is: what brand expander chips is Supermicro using, and what firmware do they load into it? About 10 years ago, some Supermicro's backplanes used vanilla Maxim VSC expanders (the former Dallas Semiconductor line), with vanilla Maxim firmware. The nice thing about those was that Maxim was pretty bug-free, and their engineering groups were friendly and easy to work with. Other backplanes used SAS chips from LSI/Broadcom (big advantage: if you use those with an LSI/Broadcom HBA and it doesn't work, you know whose fault it is), and some from PMC-Sierra (today Microsemi, used to be branded as Adaptec, and in my biased opinion those have no redeeming value).

The SAS expander is typically a SCSI device of its own (in its function as enclosure controller), and in theory you could send it a SCSI inquiry command, to determine make/model/serial, and importantly its firmware version. In the situation of a captive chip, that's probably pointless, as make/model will probably just say "Supermicro" and the model number of the backplane. Finding and fixing bugs in the expander firmware would require working with Supermicro engineering, which is difficult, and likely impossible for an individual customer.

This is also reported by FreeBSD's dmesg:
Code: 
mpr0: Firmware: 24.00.00.00. Driver: 23.00.00.00-fbsd
Click to expand...
It may be correct that LSI/Broadcom firmware 24 uses FreeBSD driver 23, but that mismatch does not make me feel warm and fuzzy.
 
You must log in or register to reply here.
Back
Top