Please help with mysterious disks problem on system with two M1015 SAS/SATA controller.
Description of a problem
There are two M1015 SAS/SATA controllers (LSI 8240 flashed to 8211-IT) placed into SuperMicro board.
Six drives are working in raidz2 (new_pool) without any problems (4 of then on mps0 and 2 on mps1).
After connecting 8 disk (old_pool1, old_pool2) from another machine, one of the disks was not recognized. Not shown in
Placing this disk in different location (controller, power, data connector) did not help.
Controllers are recognized and working without a problem. There is a difference between firmware and BSD driver. For tests, LSI drivers 16,17,18 and 19 were downloaded and placed as modules. No change. ):
However, after hundreds of test I manage to test with 2 more drives... and the magic begin...
After placing one disks on M1015 it shows up, the one "not seen" was still not seen.
After replacing "not seen" disk with new one, the new one was not seen.
After placing two new disks on M1015 both shows up, the one "not seen" was still not seen.
After placing them on motherboard controller the one "not seen" was still not seen.
After removing 2 new drives and connecting "not seen" one to motherboard controller, disk shows up, but one disk from new_pool disappears...
It behaves like the number of disks are badly reported to FreeBSD by mps driver and there is always one missing. I am not sure who blame: hardware, driver or system? Maybe it is connected with firmware and bsd driver mismatch? However, I have heard that driver 18 was skipped, but firmware 17 was compatible with driver 16. Is that right? The tests with different LSI drivers (not changing firmware) doesn't change much.
Was there a problem earlier with mps driver, which badly enumerated connected drives? If so, is the problem fixed for sure?
Is there something I do not know? What to look for? Who talk to about this problem? MAV@?
Known for sure:
- no answer on google / RTFM / this forum
- new_pool drives (same model_1, size, firmware, sata-3)
- old_pools drives (same model_2, size, firmware, sata-2)
- additional drives (same model_3, size, different firmwares, sata-3)
- disk connected directly to controllers
- all data and power connector are verified to be okay
- all disk are ok and are seen and works in other machines
- no relocated sectors, no curent pending sectors, SMART okay
-
- there is enough power for disks
- "not seen" disk is not spinning (not warm after a while and no vibration can be sensed)
Description of a problem
There are two M1015 SAS/SATA controllers (LSI 8240 flashed to 8211-IT) placed into SuperMicro board.
Six drives are working in raidz2 (new_pool) without any problems (4 of then on mps0 and 2 on mps1).
After connecting 8 disk (old_pool1, old_pool2) from another machine, one of the disks was not recognized. Not shown in
dmesg
. The old_pool2 was degraded.Placing this disk in different location (controller, power, data connector) did not help.
Controllers are recognized and working without a problem. There is a difference between firmware and BSD driver. For tests, LSI drivers 16,17,18 and 19 were downloaded and placed as modules. No change. ):
Code:
mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xf74c0000-0xf74c3fff,0xf7480000-0xf74bffff irq 16 at device 0.0 on pci1
mps0: Firmware: 17.00.01.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps1: <LSI SAS2008> port 0xd000-0xd0ff mem 0xf73c0000-0xf73c3fff,0xf7380000-0xf73bffff irq 17 at device 0.0 on pci2
mps1: Firmware: 17.00.01.00, Driver: 16.00.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
However, after hundreds of test I manage to test with 2 more drives... and the magic begin...
After placing one disks on M1015 it shows up, the one "not seen" was still not seen.
After replacing "not seen" disk with new one, the new one was not seen.
After placing two new disks on M1015 both shows up, the one "not seen" was still not seen.
After placing them on motherboard controller the one "not seen" was still not seen.
After removing 2 new drives and connecting "not seen" one to motherboard controller, disk shows up, but one disk from new_pool disappears...
It behaves like the number of disks are badly reported to FreeBSD by mps driver and there is always one missing. I am not sure who blame: hardware, driver or system? Maybe it is connected with firmware and bsd driver mismatch? However, I have heard that driver 18 was skipped, but firmware 17 was compatible with driver 16. Is that right? The tests with different LSI drivers (not changing firmware) doesn't change much.
Was there a problem earlier with mps driver, which badly enumerated connected drives? If so, is the problem fixed for sure?
Is there something I do not know? What to look for? Who talk to about this problem? MAV@?
Known for sure:
- no answer on google / RTFM / this forum
- new_pool drives (same model_1, size, firmware, sata-3)
- old_pools drives (same model_2, size, firmware, sata-2)
- additional drives (same model_3, size, different firmwares, sata-3)
- disk connected directly to controllers
- all data and power connector are verified to be okay
- all disk are ok and are seen and works in other machines
- no relocated sectors, no curent pending sectors, SMART okay
-
camcontrol devlist
always shows the same disks shown in dmesg
- there is enough power for disks
- "not seen" disk is not spinning (not warm after a while and no vibration can be sensed)
Last edited: