mpt driver not reporting all passthrough devices

[Terri_Kennedy]

Background: This all started when trying to get sysutils/smartmontools to report on the physical drives behind a mpt(4) controller. For some reason, only one of the physical drives is detected and attached to the pass(4) driver.

The hardware is a Dell PowerEdge R300 with a SAS 6/iR controller (LSI 1068E) and 2 Seagate ST373455SS SAS 15k.5 drives in a mirror set. All firmware, BIOS, etc. is up-to-date. This is under FreeBSD 8.4-PRERELEASE. Here are the relevant pieces from dmesg:

mpt0: <LSILogic SAS/SATA Adapter> port 0xec00-0xecff mem 0xdfcec000-0xdfceffff,0xdfcf0000-0xdfcfffff irq 16 at device 0.0 on pci5
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 1 Active Volume (2 Max)
mpt0: 2 Hidden Drive Members (14 Max)
mpt0:vol0(mpt0:0:0): Settings ( Member-WCE Hot-Plug-Spares High-Priority-ReSync )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0): 2 Members:
      (mpt0:1:9:0): Primary Online
      (mpt0:1:1:0): Secondary Online
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
(mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:1): Online
(mpt0:vol0:0): Physical (mpt0:0:9:0), Pass-thru (mpt0:1:1:0)
(mpt0:vol0:0): Online
(probe0:mpt0:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 
(probe0:mpt0:0:0:0): CAM status: SCSI Status Error
(probe0:mpt0:0:0:0): SCSI status: Check Condition
(probe0:mpt0:0:0:0): SCSI sense: ILLEGAL REQUEST info?:39000000 asc:0,0 (No additional sense information)
ses0 at mpt0 bus 0 scbus0 target 8 lun 0
ses0: <DP BACKPLANE 1.05> Fixed Enclosure Services SCSI-5 device 
ses0: 300.000MB/s transfers
ses0: SCSI-3 SES Device
pass2 at mpt0 bus 1 scbus1 target 0 lun 0
pass2: <SEAGATE ST373455SS S52C> Fixed Uninstalled SCSI-5 device 
pass2: 300.000MB/s transfers
pass2: Command Queueing enabled
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <Dell VIRTUAL DISK 1028> Fixed Direct Access SCSI-5 device 
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 69376MB (142082048 512 byte sectors: 255H 63S/T 8844C)

The /dev/pass2 device that's detected is the first physical member of the mirrored volume, confirmed by retrieving the drive serial numbers from the controller's BIOS and comparing with the output from smartctl -a /dev/pass2. Also, the controller's BIOS says the first physical device is "slot 0 scan order 9" and the second is "slot 1 scan order 1", which agrees with physical devices listed in the above dmesg output.

I believe the probe errors above are just the mpt(4) driver being overly chatty as I see similar things on other hardware with those controllers.

I tried booting FreeBSD 9.1 r248611 (mfsBSD) and FreeBSD HEAD r249710 (from here) and both behaved the same way as 8.4-PRERELEASE, only attaching one of the physical drives to a pass(4) device. What is interesting is that all of these FreeBSD versions "know" that there are two physical devices, as shown in this excerpt of the above dmesg output:

(mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:0): Physical (mpt0:0:9:0), Pass-thru (mpt0:1:1:0)

Moving further afield, I tried OpenBSD 5.2 (same result) and Debian 6.0.7 (Linux kernel 2.6.32). Debian did detect both physical members and assigned them sg device names. So it doesn't seem to be a hardware problem.

 

[Terri_Kennedy]

I experimented with a number of different configurations - 2 drives, 4 drives, RAID 0, RAID 1, hot spares, exported raw drives, etc. and I now know what the mpt(4) driver is doing, but I don't know why.

If there is a single RAID volume present (any number of drives / RAID level / hot spares), there will be one missing pass(4) device. So a RAID volume with 4 members (or 2 members and 2 hot spares) will report 3 pass(4) devices.

If no RAID volume is present (in which case the controller exports the raw drives), the correct number of pass(4) devices appear.

I was not able to test what happens when there are 2 RAID volumes on the controller (BIOS restriction), but my guess is that 2 pass(4) devices will be missing.

Again, this issue seems to happen across FreeBSD versions (8, 9, and HEAD) and on OpenBSD, perhaps indicating a common driver ancestry. Debian Linux does detect the correct number of passthru devices (which it calls sg).

Any developers want to take a stab at this here, or should I file a PR on this?

 

[Terri_Kennedy]

Any developers want to take a stab at this here, or should I file a PR on this?

Bueller? ... Anyone? ...

I did some more testing and releases before 8.3-RELEASE do not even show the pass(4) device for one of the underlying physical devices. I copied the entire /sys/dev/mpt tree from 8.2-RELEASE to an 8-STABLE (8.4-PRERELEASE, give or take, net/svnup'd this evening) source tree and built a new kernel. That kernel also does not see the pass(4) device for the physical drive. This rules out the issue being somewhere in cam(4) or elsewhere in the kernel, as the rest of the kernel was unchanged.

I was not able to experiment with mix-and-match of /sys/dev/mpt components between 8.2 and 8.4 as there are a number of dependencies between them which prevents trying one file at a time.