Other SAS Multipath problem

T

Tualua

Hello! I'm trying to get multipath working without any success.

My hardware is:
Server: HPE Proliant DL360p Gen9
SAS HBA: LSI 9300-8e (latest firmware)
External enclosure: HP D2600 SAS2 JBOD Dual Controller
HDDs: NL-SAS Dual-port SEAGATE ST6000NM0034 x10
SSD: Stec ZeusRAM 8GB
OS: FreeBSD 12.1, fresh install +

Each port of HBA is connected to different JBOD controller
The problem is that the OS can see only one path no matter what I do.

If I disconnect one cable from one JBOD controller OS can't see disks and enclosure at all. If I reboot system after that OS can see disks again on second cable.
HBA is OK, I double checked. I can see all 22 paths for disks on BIOS boot screen and LSI configuration utility.

In FreeBSD:
camcontrol devlist - 11 disks.
Code: 
<HP MB6000JEFND HPD5>              at scbus0 target 0 lun 0 (pass0,da0)
<HP MB6000JEFND HPD3>              at scbus0 target 1 lun 0 (pass1,da1)
<SEAGATE ST6000NM0034 MS85>        at scbus0 target 2 lun 0 (pass2,da2)
<SEAGATE ST6000NM0034 MS84>        at scbus0 target 3 lun 0 (pass3,da3)
<SEAGATE ST6000NM0034 MS2E>        at scbus0 target 4 lun 0 (pass4,da4)
<SEAGATE ST6000NM0034 MS85>        at scbus0 target 5 lun 0 (pass5,da5)
<SEAGATE ST6000NM0034 MS85>        at scbus0 target 6 lun 0 (pass6,da6)
<SEAGATE ST6000NM0034 MS84>        at scbus0 target 7 lun 0 (pass7,da7)
<SEAGATE ST6000NM0034 MS2E>        at scbus0 target 8 lun 0 (pass8,da8)
<SEAGATE ST6000NM0034 MS85>        at scbus0 target 10 lun 0 (pass9,da9)
<STEC ZeusRAM C025>                at scbus0 target 11 lun 0 (pass10,da10)
<HP D2600 SAS AJ940A 0150>         at scbus0 target 24 lun 0 (ses0,pass11)
<HP RAID 0 OK>                     at scbus1 target 0 lun 0 (pass12,da11)
<HP Virtual DVD-ROM \0000.0>       at scbus6 target 0 lun 0 (pass13,cd0)
<kdata KDATA 1.00>                 at scbus7 target 0 lun 0 (da12,pass14)

sas3ircu 0 display - 11 disks.

dmesg | grep mpr - 11 disks
Code: 
mpr0: <Avago Technologies (LSI) SAS3008> port 0x6000-0x60ff mem 0xf7ef0000-0xf7efffff,0xf7e80000-0xf7ebffff irq 26 at device 0.0 numa-domain 0 on pci1
mpr0: Firmware: 16.00.10.00, Driver: 23.00.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 13
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 14
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 15
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 16
mpr0: At enclosure level 0 and connector name (    )
mpr0: nvd0: 1831420MB (468843606 4096 byte sectors)
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 18
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 19
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 20
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 21
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 23
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <401<SspTarg>,End Device> <6.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 24
mpr0: At enclosure level 0 and connector name (    )
mpr0: Found device <4441<SspInit,SspTarg,SepDev>,End Device> <6.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (    )
da1 at mpr0 bus 0 scbus0 target 1 lun 0
da0 at mpr0 bus 0 scbus0 target 0 lun 0
da10 at mpr0 bus 0 scbus0 target 11 lun 0
da3 at mpr0 bus 0 scbus0 target 3 lun 0
da2 at mpr0 bus 0 scbus0 target 2 lun 0
da5 at mpr0 bus 0 scbus0 target 5 lun 0
da6 at mpr0 bus 0 scbus0 target 6 lun 0
da8 at mpr0 bus 0 scbus0 target 8 lun 0
da9 at mpr0 bus 0 scbus0 target 10 lun 0
da4 at mpr0 bus 0 scbus0 target 4 lun 0
da7 at mpr0 bus 0 scbus0 target 7 lun 0
ses0 at mpr0 bus 0 scbus0 target 24 lun 0
All log files are attached to this post.

I tried to boot to Centos7-Live and it can see all disks at all paths (log files also attached):

lsscsi
Code: 
[0:0:0:0]    cd/dvd  HP       Virtual DVD-ROM        /dev/sr0
[3:0:0:0]    disk    kdata    KDATA            1.00  /dev/sda
[4:0:0:0]    storage HP       P420i            8.32  -    
[4:1:0:0]    disk    HP       LOGICAL VOLUME   8.32  /dev/sdb
[5:0:0:0]    disk    HP       MB6000JEFND      HPD5  /dev/sdc
[5:0:1:0]    disk    HP       MB6000JEFND      HPD3  /dev/sdd
[5:0:2:0]    disk    SEAGATE  ST6000NM0034     MS85  /dev/sde
[5:0:3:0]    disk    SEAGATE  ST6000NM0034     MS84  /dev/sdf
[5:0:4:0]    disk    SEAGATE  ST6000NM0034     MS2E  /dev/sdg
[5:0:5:0]    disk    SEAGATE  ST6000NM0034     MS85  /dev/sdh
[5:0:6:0]    disk    SEAGATE  ST6000NM0034     MS85  /dev/sdi
[5:0:7:0]    disk    SEAGATE  ST6000NM0034     MS84  /dev/sdj
[5:0:8:0]    disk    SEAGATE  ST6000NM0034     MS2E  /dev/sdk
[5:0:9:0]    disk    SEAGATE  ST6000NM0034     MS85  /dev/sdl
[5:0:10:0]   disk    STEC     ZeusRAM          C025  /dev/sdm
[5:0:11:0]   enclosu HP       D2600 SAS AJ940A 0150  -    
[5:0:12:0]   disk    HP       MB6000JEFND      HPD5  /dev/sdn
[5:0:13:0]   disk    HP       MB6000JEFND      HPD3  /dev/sdo
[5:0:14:0]   disk    SEAGATE  ST6000NM0034     MS85  /dev/sdp
[5:0:15:0]   disk    SEAGATE  ST6000NM0034     MS84  /dev/sdq
[5:0:16:0]   disk    SEAGATE  ST6000NM0034     MS2E  /dev/sdr
[5:0:17:0]   disk    SEAGATE  ST6000NM0034     MS85  /dev/sds
[5:0:18:0]   disk    SEAGATE  ST6000NM0034     MS85  /dev/sdt
[5:0:19:0]   disk    SEAGATE  ST6000NM0034     MS84  /dev/sdu
[5:0:20:0]   disk    SEAGATE  ST6000NM0034     MS2E  /dev/sdv
[5:0:21:0]   disk    SEAGATE  ST6000NM0034     MS85  /dev/sdw
[5:0:22:0]   disk    STEC     ZeusRAM          C025  /dev/sdx
[5:0:23:0]   enclosu HP       D2600 SAS AJ940A 0150  -

What am I doing wrong?
 

Attachments

  • linux-ircu.txt
    19.3 KB · Views: 107
  • linux-lsscsi.txt
    1.7 KB · Views: 128
  • linux-mpath.txt
    3.2 KB · Views: 111
  • devlist.txt
    1.1 KB · Views: 87
  • dmesg.txt
    28 KB · Views: 116
Screenshots from AVAGO Configuration Utility
2020-08-20 (4).png2020-08-20 (3).png2020-08-20 (2).png2020-08-20 (1).png2020-08-20.png
 
Tualua said:
What do you think? :) Of course they are NL-SAS dual port
Click to expand...
I just asked to get the obvious out of the way. I have managed to get multipathing working with HBA in IT mode, with another card, using a different driver (mps), so there may be an issue with that model's driver you are facing?
 
Upgrade the firmware on your enclosure, and on the HBAs. While you are at it, running a firmware upgrade on the disks wouldn't hurt either, but it's unlikely to be the problem.

Your disks are Seagate models. Please investigate whether the HP enclosure has limitations to check the disks for vendor and model number; it might only be willing to function completely normally with certain disks installed (for example HP disks). You might get help on that from HP customer support, assuming you have a service contract. The same applies to the HBA: It might be an HP-branded LSI card, which knows how to check for certain disk models.
 
Enclosure is on latest firmware and this is "dumb" enclosure which accepts any disks, even SATA. HBA is generic LSI, not branded. As I said before I have no problems under Linux - it can see all disks and all paths and survives disconnects of 1 of 2 cables without any problems and Avago Configuration Utility also can see all disks. I'm sure this is not a hardware problem.

Googling for "freebsd mpr multipath" gave me the following:

229371 – LSI 9300-8e Controller JBOD Multipath not functioning

bugs.freebsd.org bugs.freebsd.org

Bug #35988: mpr(4) not enumerating some disks - FreeNAS - iXsystems & FreeNAS Redmine

Redmine
redmine.ixsystems.com redmine.ixsystems.com

I think there is definitely a bug in the mpr driver
 
Tualua said:
Nope, not working. Switched to 9207-8e - everything is fine. Definitely a bug in mpr.
Click to expand...
It's at least good you've pinpointed the issue, something that makes problem reporting easier. Although it does nothing to help you right now, please report it, so that it may get fixed in the future!
 
You must log in or register to reply here.
Back
Top