Hi, All! Today, while upgrading FreeBSD 11.2 to 12.2 we had an issue, where drives attached to LSI 3108 controller are detected, but then detached an gone during boot time (seems like everything happens before init(8))? The machine have since been rolled-back to 11.2 and is doing just fine. From dmesg(8):
Selected output from
Selected output from
IIRC the same happened with 12.1, as well. Any ideas what could've gone wrong or how to deal with that? Thanks!
Code:
...
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 07.709.04.00-fbsd
mrsas0: <AVAGO Invader SAS Controller> port 0xe000-0xe0ff mem 0xfb300000-0xfb30ffff,0xfb200000-0xfb2fffff irq 26 at device 0.0 numa-domain 0 on pci2
mrsas0: FW now in Ready state
mrsas0: Using MSI-X with 24 number of vectors
mrsas0: FW supports <96> MSIX vector,Online CPU 24 Current MSIX <24>
mrsas0: max sge: 0x46, max chain frame size: 0x400, max fw cmd: 0x39f
mrsas0: Issuing IOC INIT command to FW.
mrsas0: IOC INIT response received from FW.
mrsas0: System PD created target ID: 0x8
...
mrsas0: System PD created target ID: 0x1c
mrsas0: max_fw_cmds: 927 max_scsi_cmds: 911
mrsas0: MSI-x interrupts setup success
mrsas0: mrsas_ocr_thread
...
da1 at mrsas0 bus 1 scbus1 target 18 lun 0
da1: <ATA INTEL SSDSC2KB03 0110> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number ***
da1: 150.000MB/s transfers
da1: 3662830MB (7501476528 512 byte sectors)
da4 at mrsas0 bus 1 scbus1 target 21 lun 0
da4: <ATA INTEL SSDSC2KB03 0110> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number ***
da4: 150.000MB/s transfers
da4: 3662830MB (7501476528 512 byte sectors)
ses0 at mrsas0 bus 1 scbus1 target 8 lun 0
ses0: <LSI SAS3x28 0601> Fixed Enclosure Services SPC-3 SCSI device
ses0: 150.000MB/s transfers
ses0: SES Device
da0 at mrsas0 bus 1 scbus1 target 17 lun 0
da0: <ATA SAMSUNG MZ7LM960 104Q> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number ***
da0: 150.000MB/s transfers
da0: 915715MB (1875385008 512 byte sectors)
da0: quirks=0x8<4K>
...
da7 at mrsas0 bus 1 scbus1 target 24 lun 0
da7: <ATA SAMSUNG MZ7LM960 104Q> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number ***
da7: 150.000MB/s transfers
da7: 915715MB (1875385008 512 byte sectors)
da7: quirks=0x8<4K>
ses1 at ahciem0 bus 0 scbus8 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
da3 at mrsas0 bus 1 scbus1 target 20 lun 0
da3: <ATA SAMSUNG MZ7LM960 104Q> s/n *** detached
da8 at mrsas0 bus 1 scbus1 target 25 lun 0
da8: <ATA SAMSUNG MZ7LM960 204Q> s/n *** detached
ses1: ada0,pass13 in 'Slot 00', SATA Slot: scbus2 target 0
da4 at mrsas0 bus 1 scbus1 target 21 lun 0
da4: <ATA INTEL SSDSC2KB03 0110> s/n *** detached
ses0: da0,pass1 in 'Slot00', SAS Slot: 1 phys at slot 0
da0 at mrsas0 bus 1 scbus1 target 17 lun 0
da0: <ATA SAMSUNG MZ7LM960 104Q> s/n *** detached
ses0: phy 0: SATA device
ses0: phy 0: parent 500304801ee1dbbf addr 500304801ee1db80
...
ses0: da4,pass5 in 'Slot01', SAS Slot: 1 phys at slot 1
ses0: phy 0: SATA device
da2 at mrsas0 bus 1 scbus1 target 19 lun 0
da2: <ATA SAMSUNG MZ7LM960 104Q> s/n *** detached
ses0: phy 0: parent 500304801ee1dbbf addr 500304801ee1db81
g_access(961): provider da0 has error 6 set
ses0: da1 in 'Slot02', SAS Slot: 1 phys at slot 2
g_access(961): provider da0 has error 6 set
g_access(961): provider da0 has error 6 set
da7 at mrsas0 bus 1 scbus1 target 24 lun 0
da7: <ATA SAMSUNG MZ7LM960 104Q> s/n *** detached
ses0: phy 0: SATA device
ses0: phy 0: parent 500304801ee1dbbf addr 500304801ee1db82
ses0: da5 in 'Slot03', SAS Slot: 1 phys at slot 3
g_access(961): provider da0 has error 6 set
(da3:mrsas0:1:20:0): Periph destroyed
(da8:mrsas0:1:25:0): Periph destroyed
(da4:mrsas0:1:21:0): Periph destroyed
(da5:mrsas0:1:22:0): Periph destroyed
(da1:mrsas0:1:18:0): Periph destroyed
(da6:mrsas0:1:23:0): Periph destroyed
...
arc4random: no preloaded entropy cache
...
ioat0: <BDX IOAT Ch0> mem 0xfb72c000-0xfb72ffff irq 31 at device 4.0 numa-domain 0 on pci1
ioat0: Capabilities: 2f7<PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
...
lo0: link state changed to UP
ix0: link state changed to UP
...
Security policy loaded: MAC/ntpd (mac_ntpd)Selected output from
storcli /c0 show all:
Code:
CLI Version = 007.1017.0000.0000 May 10, 2019
Operating system = FreeBSD 12.2-RELEASE-p3
Model = LSI 3108 MegaRAID
SAS Address = 500304801e065400
PCI Address = 00:01:00:00
Version :
=======
Firmware Package Build = 24.5.0-0015
Firmware Version = 4.250.00-3569
Bios Version = 6.19.05.0_4.16.08.00_0x06080500
Ctrl-R Version = 5.06-0003
Preboot CLI Version = 01.07-05:#%0000
NVDATA Version = 3.1406.00-0075
Boot Block Version = 3.02.00.00-0001
Driver Name = mrsas
Driver Version = 07.709.04.00-fbsd
Status :
======
Controller Status = Optimal
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
ECC Bucket Count = 0
Any Offline VD Cache Preserved = No
BBU Status = 0
PD Firmware Download in progress = No
Support PD Firmware Download = No
Lock Key Assigned = No
Failed to get lock key on bootup = No
Lock key has not been backed up = No
Bios was not detected during boot = No
Controller must be rebooted to complete security operation = No
A rollback operation is in progress = No
At least one PFK exists in NVRAM = No
SSC Policy is WB = No
Controller has booted into safe mode = No
Controller shutdown required = No
HwCfg :
=====
ChipRevision = C0
BatteryFRU = N/A
Front End Port Count = 0
Backend Port Count = 8
BBU = Present
Alarm = On
Serial Debugger = Present
NVRAM Size = 32KB
Flash Size = 16MB
On Board Memory Size = 2048MB
CacheVault Flash Size = 8.000 GB
TPM = Absent
Upgrade Key = Absent
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Upgradable CPLD = Absent
Upgradable PSOC = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 0
ROC temperature(Degree Celsius) = 60
Capabilities :
============
Supported Drives = SAS, SATA
RAID Level Supported = RAID0, RAID1(2 or more drives), RAID5, RAID6, RAID00, RAID10(2 or more drives per span), RAID50, RAID60
Enable JBOD = Yes
JBOD Drives = 12
JBOD LIST :
=========
----------------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------------
8:0 17 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:1 21 JBOD - 3.492 TB SATA SSD N N 512B INTEL SSDSC2KB038T8 U -
8:2 18 JBOD - 3.492 TB SATA SSD N N 512B INTEL SSDSC2KB038T8 U -
8:3 22 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:4 19 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:5 24 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:6 20 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:7 23 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:8 26 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:9 27 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:10 25 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
8:11 28 JBOD - 894.252 GB SATA SSD N N 512B SAMSUNG MZ7LM960HMJP-00005 U -
----------------------------------------------------------------------------------------Selected output from
storcli /c0 show events:
Code:
seqNum: 0x000025ed
Time: Wed Mar 10 11:06:42 2021
Code: 0x00000071
Class: 0
Locale: 0x02
Event Description: Unexpected sense: PD 13(e0x08/s4) Path 500304801ee1db84, CDB: 00 00 00 00 00 00, Sense: b/00/00
Event Data:
===========
Device ID: 19
Enclosure Index: 8
Slot Number: 4
CDB Length: 6
CDB Data:
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 Sense Length: 18
Sense Data:
0070 0000 000b 0000 0000 0000 0000 000a 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
seqNum: 0x000025ee
Time: Wed Mar 10 11:06:42 2021
Code: 0x00000071
Class: 0
Locale: 0x02
Event Description: Unexpected sense: PD 18(e0x08/s5) Path 500304801ee1db85, CDB: 00 00 00 00 00 00, Sense: b/00/00
Event Data:
===========
Device ID: 24
Enclosure Index: 8
Slot Number: 5
CDB Length: 6
CDB Data:
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 Sense Length: 18
Sense Data:
0070 0000 000b 0000 0000 0000 0000 000a 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000IIRC the same happened with 12.1, as well. Any ideas what could've gone wrong or how to deal with that? Thanks!