Solved Fibre Channel Target mode

MurrayRothbard · Monday at 6:39 PM

Hi, I am trying to set up a FreeBSD Fibre Channel target.

/etc/ctl.conf

Code:

lun internal-firewall {
    backend block
    path /dev/zvol/storage/vm/internal-firewall
}

target naa.210034800d71282a {
    alias xcpng-hypervisor
    auth-type none
    port isp0
    lun 0 internal-firewall
}

ctladm portlist

Code:

Port Online Frontend Name     pp vp
0    YES    camsim   camsim   0  0  naa.5000000e0aa6af01
1    YES    ioctl    ioctl    0  0  
2    YES    tpc      tpc      0  0  
3    YES    camtgt   isp0     0  0  naa.210034800d71282a
4    YES    camtgt   isp1     0  0  naa.210034800d71282b

dmesg

Code:

isp0: <Qlogic ISP 2722 PCI FC Adapter> mem 0xe0505000-0xe0505fff,0xe0502000-0xe0503fff,0xe0400000-0xe04fffff at device 0.0 on pci1
isp0: Primary aux image not addressed
isp0: Secondary aux image not addressed
isp1: <Qlogic ISP 2722 PCI FC Adapter> mem 0xe0504000-0xe0504fff,0xe0500000-0xe0501fff,0xe0300000-0xe03fffff at device 0.1 on pci1
isp1: Primary aux image not addressed
isp1: Secondary aux image not addressed
ctlfe_onoffline: isp0 current WWNN 0x200034800d71282a
ctlfe_onoffline: isp0 current WWPN 0x210034800d71282a
isp0: Setting Channel 0 wwns to 0x210034800d71282a 0x210034800d71282a
ctlfe_onoffline: isp0 new WWNN 0x210034800d71282a
ctlfe_onoffline: isp0 new WWPN 0x210034800d71282a
ctlfe_onoffline: isp0 (path id 0) target role enable succeeded
ctlfeasync: WWPN 0x21000024ff1d7dc8 port 0x0000e8 path 0 target 0 arrived
ctlfeasync: WWPN 0x21000024ff1d7dc8 port 0x0000e8 path 0 target 0 left

On the Linux initiator, I get this message:
qla2xxx [0000:05:00.0]-705d:0: Remote port 21 00 34 80 0d 71 28 2a is not coming back

When I check on the target side, if a LUN is exposed:
ctladm lunlist
Nothing

ctladm devlist

Code:

LUN Backend       Size (Blocks)   BS Serial Number    Device ID

When I restart the ctld service
service ctld restart:

ctladm devlist

Code:

LUN Backend       Size (Blocks)   BS Serial Number    Device ID       
  0 block             251658240  512 MYSERIAL0000     MYDEVID0000

ctladm lunlist

Code:

(7:1:0/0): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device

But now I have this iSCSI frontend
ctladm portlist

Code:

Port Online Frontend Name     pp vp
0    YES    camsim   camsim   0  0  naa.5000000e0aa6af01
1    YES    ioctl    ioctl    0  0  
2    YES    tpc      tpc      0  0  
3    YES    camtgt   isp0     0  0  naa.210034800d71282a
4    YES    camtgt   isp1     0  0  naa.210034800d71282b
5    YES    iscsi    iscsi    256 1  naa.210034800d71282a,t,0x0100

From my understanding, the iSCSI frontend should not be there. But I do not understand why and how it got there and why it is created/used even if I specified the port with isp0 and the WWPN as the target.

I have tested this setup with the same Fedora initiator and a Fedora target, and it worked. But after I was back in FreeBSD, suddenly ctl could not see the ports at all.

ctladm portlist

Code:

Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.50000008a15d8b01
1    YES    ioctl    ioctl    0  0  
2    YES    tpc      tpc      0  0

pciconf -lvV

Code:

isp0@pci0:1:0:0:    class=0x0c0400 rev=0x01 hdr=0x00 vendor=0x1077 device=0x2261 subvendor=0x1077 subdevice=0x029c
    vendor     = 'QLogic Corp.'
    device     = 'ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter'
    class      = serial bus
    subclass   = Fibre Channel
    VPD ident  = 'QLogic 16Gb 2-port FC to PCIe Gen3 x8 Adapter'
    VPD ro PN  = 'QLE2692'
    VPD ro SN  = 'RFD2019J37118'
    VPD ro EC  = 'BK3210407-05  D'
    VPD ro V9  = '010159'
isp1@pci0:1:0:1:    class=0x0c0400 rev=0x01 hdr=0x00 vendor=0x1077 device=0x2261 subvendor=0x1077 subdevice=0x029c
    vendor     = 'QLogic Corp.'
    device     = 'ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter'
    class      = serial bus
    subclass   = Fibre Channel
    VPD ident  = 'QLogic 16Gb 2-port FC to PCIe Gen3 x8 Adapter'
    VPD ro PN  = 'QLE2692'
    VPD ro SN  = 'RFD2019J37118'
    VPD ro EC  = 'BK3210407-05  D'
    VPD ro V9  = '010159'

sysctl dev.isp

Code:

dev.isp.1.fw_version_run: 9.12.0
dev.isp.1.fw_version_ispfw: 9.12.0
dev.isp.1.fw_version_flash: 8.5.44
dev.isp.1.use_gff_id: 1
dev.isp.1.use_gft_id: 1
dev.isp.1.topo: 0
dev.isp.1.loopstate: 0
dev.isp.1.fwstate: 0
dev.isp.1.linkstate: 0
dev.isp.1.speed: 0
dev.isp.1.role: 2
dev.isp.1.gone_device_time: 30
dev.isp.1.loop_down_limit: 60
dev.isp.1.wwpn: 2377958327837599787
dev.isp.1.wwnn: 2305900733799671851
dev.isp.1.%iommu: rid=0x101
dev.isp.1.%parent: pci1
dev.isp.1.%pnpinfo: vendor=0x1077 device=0x2261 subvendor=0x1077 subdevice=0x029c class=0x0c0400iscsi
dev.isp.1.%location: slot=0 function=1 dbsf=pci0:1:0:1 handle=\_SB_.PCI0.GPP0.HDAU
dev.isp.1.%driver: isp
dev.isp.1.%desc: Qlogic ISP 2722 PCI FC Adapter
dev.isp.0.fw_version_run: 9.12.0
dev.isp.0.fw_version_ispfw: 9.12.0
dev.isp.0.fw_version_flash: 8.5.44
dev.isp.0.use_gff_id: 1
dev.isp.0.use_gft_id: 1
dev.isp.0.topo: 0
dev.isp.0.loopstate: 0
dev.isp.0.fwstate: 0
dev.isp.0.linkstate: 0
dev.isp.0.speed: 0
dev.isp.0.role: 2
dev.isp.0.gone_device_time: 30
dev.isp.0.loop_down_limit: 60
dev.isp.0.wwpn: 2377958327837599786
dev.isp.0.wwnn: 2305900733799671850
dev.isp.0.%iommu: rid=0x100
dev.isp.0.%parent: pci1
dev.isp.0.%pnpinfo: vendor=0x1077 device=0x2261 subvendor=0x1077 subdevice=0x029c class=0x0c0400
dev.isp.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.GPP0.VGA_
dev.isp.0.%driver: isp
dev.isp.0.%desc: Qlogic ISP 2722 PCI FC Adapter
dev.isp.%parent:

dmesg

Code:

isp0: <Qlogic ISP 2722 PCI FC Adapter> mem 0xe0505000-0xe0505fff,0xe0502000-0xe0503fff,0xe0400000-0xe04fffff at device 0.0 on pci1
isp0: Primary aux image not addressed
isp0: Secondary aux image not addressed
isp1: <Qlogic ISP 2722 PCI FC Adapter> mem 0xe0504000-0xe0504fff,0xe0500000-0xe0501fff,0xe0300000-0xe03fffff at device 0.1 on pci1
isp1: Primary aux image not addressed
isp1: Secondary aux image not addressed

robroy · Monday at 8:00 PM

MurrayRothbard, we're twins; I'm using target mode with the quad-port edition of that same HBA (QLE2694) in 14.3-RELEASE.

My QLE2694 was running firmware level 8.03.05. And at that firmware level, isp(4) became unhappy and stalled the host during boot (14.0-RELEASE was fine in this regard; the change that caused this was introduced in 14.1-RELEASE). After having used the QLogic tools to update to 9.15.01, that problem went away.

Since your target ports aren't showing up consistently, do you think you could try updating your HBA's firmware? I get that you're using what was loaded at runtime by ispfw(4), but I've suffered through exasperating evils rooted in old FC HBA firmware too many times over the years not to bring stuff up to date.

Even with the current firmware, here's what I had to do to get things working:

These lines were necessary in /boot/device.hints to get the ports to actually enter target mode. Without these lines, my initiators saw nothing despite everything looking OK on FreeBSD. Note that the 2 value is supposed to hard-wire isp(4) to initiator mode, according to the man page--but it actually put my card in to target mode... ~BoInG~! I was rewarded by this discovery after a mere sixty billion hours of tinkering.
Code:
```
hint.isp.0.role="2"
hint.isp.1.role="2"
hint.isp.2.role="2"
hint.isp.3.role="2"
```

In case it's a handy reference, here's my working /etc/ctl.conf. You can see that ports 0 and 3 have the same LUs presented on them, while ports 1 and 2 differ.

Code:

target naa.210034800d60ae44 {
    port isp0
    lun 0 {path /dev/zvol/s/fsE}
    lun 1 {path /dev/zvol/r/fsF}
    lun 2 {path /dev/zvol/x/exosA}
}
target naa.210034800d60ae45 {
    port isp1
    lun 0 {path /dev/zvol/r/gem2-w2k22}
}
target naa.210034800d60ae46 {
    port isp2
    lun 0 {path /dev/zvol/s/gem3-w2k22}
}
target naa.210034800d60ae47 {
    port isp3
    lun 0 {path /dev/zvol/s/fsE}
    lun 1 {path /dev/zvol/r/fsF}
    lun 2 {path /dev/zvol/x/exosA}
}

And here are a few outputs from my working system.

By the way, I have those iSCSI lines too; I guess they look odd but I didn't investigate (since my stuff worked).

Code:

root@donut:~ # ctladm devlist
LUN Backend       Size (Blocks)   BS Serial Number    Device ID    
  0 block            8589934592  512 MYSERIAL0000     MYDEVID0000  
  1 block            8589934592  512 MYSERIAL0001     MYDEVID0001  
  2 block           21474836480  512 MYSERIAL0002     MYDEVID0002  
  3 block            1048576000  512 MYSERIAL0003     MYDEVID0003  
  4 block            1048576000  512 MYSERIAL0004     MYDEVID0004  
  5 block            8589934592  512 MYSERIAL0005     MYDEVID0005  
  6 block            8589934592  512 MYSERIAL0006     MYDEVID0006  
  7 block           21474836480  512 MYSERIAL0007     MYDEVID0007  
root@donut:~ # ctladm lunlist
(7:1:0/0): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:1/1): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:2/2): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:3/3): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:4/4): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:5/5): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:6/6): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
(7:1:7/7): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
root@donut:~ # ctladm portlist
Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.50000004615ba701
1    YES    ioctl    ioctl    0  0
2    YES    tpc      tpc      0  0
3    YES    camtgt   isp0     0  0  naa.210034800d60ae44
4    YES    camtgt   isp1     0  0  naa.210034800d60ae45
5    YES    camtgt   isp2     0  0  naa.210034800d60ae46
6    YES    camtgt   isp3     0  0  naa.210034800d60ae47
7    YES    iscsi    iscsi    256 1  naa.210034800d60ae44,t,0x0100
8    YES    iscsi    iscsi    256 2  naa.210034800d60ae45,t,0x0100
9    YES    iscsi    iscsi    256 3  naa.210034800d60ae46,t,0x0100
10   YES    iscsi    iscsi    256 4  naa.210034800d60ae47,t,0x0100
root@donut:~ #

Starcat · Monday at 9:23 PM

FWIW, I just setup a parallel SCSI target with 14.3 and got the same iSCSI nonsense:

Code:

$ sudo ctladm portlist -v
Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.500000033d7b1701
  Target: naa.500000033d7b1700
  All LUNs mapped
      port_type=8
1    YES    ioctl    ioctl    0  0
  All LUNs mapped
      port_type=4
2    YES    tpc      tpc      0  0
  All LUNs mapped
      port_type=8
3    YES    camtgt   ahc0     0  0
  LUN 0: 0
      port_type=2
4    YES    iscsi    iscsi    256 1  eui.943d07ce77ecc404,t,0x0100
  Target: eui.943d07ce77ecc404
  LUN 0: 0
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=eui.943d07ce77ecc404
      ctld_portal_group_name=default
      cfiscsi_portal_group_tag=256

If that wasn't weird enough, I received an error message on the console saying that my SCSI HBA didn't have a valid WWNN/WWPN!

MurrayRothbard · Monday at 10:46 PM

I upgraded the firmware of the QLE2692 with the QConvergeConsoleCLI.

Code:

--------------------------------------------------------------------------------
Host Name                      : localhost.localdomain
PCI Location                   : PCI bus 1 device 0 function 0
PCI Location                   : PCI bus 1 device 0 function 1
HBA Model                      : QLE2692
Device ISP ID                  : ISP2261
Chip Revision                  : 0x1(A0)
HBA Description                : QLogic QLE2692 Dual Port 16Gb FC to PCIe Gen3 x8 Adapter
HBA Alias                      :
Serial Number                  : RFD2019J37118
Driver Version                 : 10.02.09.400-k
FC Bios Version                : 3.68.00
FC FCode Version               : 4.11.00
FC EFI Version                 : 7.32.00
FC Firmware Version            : 9.15.01
MPI Firmware Version           : 3.03.09
PEP Firmware Version           : 2.01.30
PEP SoftROM Version            : 2.00.11
PEP Brd Cfg Version            : 2.00.01
Firmware Preload Area Version  : 2.00.06
FC Brd Cfg Version             : 2.00.04
Flash Image Version            : 10.04.04
Family Firmware Version        : N/A
Firmware Update Lockdown       : N/A
--------------------------------------------------------------------------------

I did not modify /etc/ctl.conf

But I did add the device hints to /boot/device.hints

Code:

hint.isp.0.role="2"
hint.isp.1.role="2"

It also did not work for me until I added these device hints.

I still have the iSCSI frontend
ctladm portlist -v

Code:

Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.5000000d45a08701
  Target: naa.5000000d45a08700
  All LUNs mapped
      port_type=8
1    YES    ioctl    ioctl    0  0 
  All LUNs mapped
      port_type=4
2    YES    tpc      tpc      0  0 
  All LUNs mapped
      port_type=8
3    YES    camtgt   isp0     0  0  naa.210034800d71282a
  Target: naa.210034800d71282a
  Initiator 0: naa.21000024ff1d7dc8
  LUN 0: 0
      port_type=1
4    NO     camtgt   isp1     0  0 
  All LUNs mapped
      port_type=1
5    YES    iscsi    iscsi    256 1  naa.210034800d71282a,t,0x0100
  Target: naa.210034800d71282a
  LUN 0: 0
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=naa.210034800d71282a
      ctld_portal_group_name=default
      cfiscsi_portal_group_tag=256
      cfiscsi_target_alias=xcpng-hypervisor

What I find odd is that the WWNN is set to the WWPN by the driver
dmesg

Code:

isp0: <Qlogic ISP 2722 PCI FC Adapter> mem 0x7ff0505000-0x7ff0505fff,0x7ff0502000-0x7ff0503fff,0x7ff0400000-0x7ff04fffff at device 0.0 on pci1
isp0: Chan 0 setting role to 0x2
isp0: Primary aux image not addressed
isp0: Secondary aux image not addressed
isp1: <Qlogic ISP 2722 PCI FC Adapter> mem 0x7ff0504000-0x7ff0504fff,0x7ff0500000-0x7ff0501fff,0x7ff0300000-0x7ff03fffff at device 0.1 on pci1
isp1: Chan 0 setting role to 0x2
isp1: Primary aux image not addressed
isp1: Secondary aux image not addressed
ctlfe_onoffline: isp0 current WWNN 0x200034800d71282a
ctlfe_onoffline: isp0 current WWPN 0x210034800d71282a
isp0: Setting Channel 0 wwns to 0x210034800d71282a 0x210034800d71282a
ctlfe_onoffline: isp0 new WWNN 0x210034800d71282a
ctlfe_onoffline: isp0 new WWPN 0x210034800d71282a
ctlfe_onoffline: isp0 (path id 0) target role enable succeeded
ctlfeasync: WWPN 0x21000024ff1d7dc8 port 0x0000e8 path 0 target 0 arrived

But it is working now, so I am happy. Even if either the driver or documentation seems a bit dodgy.

Thank you very much robroy for the help

Solved Fibre Channel Target mode

MurrayRothbard

robroy

Starcat

MurrayRothbard