Other CTL HA ALUA - How-To

PershingDriver

PershingDriver

Hi, I have a SuperMicro based FreeBSD cluster consisting of two servers and four SAS attached disk shelves. Each server has two Qlogic 2672 HBA in target mode.

I want to configure CTL_HA in mode 2 to support ALUA using 10gigE port for the cluster interconnect but I cannot find any documentation on how to configure the interconnect.
 
making some progress on this

in sysctl.conf
Code: 
kern.cam.ctl.debug=2
kern.cam.ctl.ha_peer="connect <ipaddress:port>"
kern.cam.ctl.ha_role=1 # 0 for primary ; 1 for secondary
into /boot/loader.conf (sysctl stated tunables go in /boot/loader.conf)
Code: 
kern.cam.ctl.ha_id=2
kern.cam.ctl.ha_mode=2
not needed in either
Code: 
kern.cam.ctl.ha_link  # reports status of link
however now hitting this message
Code: 
sysctl: unknown oid 'kern.cam.ctl.ha_peer'
Without ha_peer being defined the cluster interconnect cannot be configured.

Any ideas... am I missing a package? .. .

FYI I am presenting LUNs over FC from this array so CTL is installed and configured.
 
OK, after applying r304737 and editing /boot/loader.conf and /etc/sysctl.conf I now get the cluster interconnect connection to establish

Server 1 config (Primary Node)
Code: 
# cat /boot/loader.conf
geom_mirror_load="YES"
geom_label_load="YES"
hw.memtest.tests=0
geom_multipath_load="YES"
kern.geom.label.disk_ident.enable=0
kern.cam.ctl.ha_id=1
kern.cam.ctl.ha_mode=2

# cat /etc/sysctl.conf
# $FreeBSD: releng/10.3/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $

kern.cam.ctl.debug=2
kern.cam.ctl.ha_peer="connect 172.16.1.3:7940"  #directly connected cable no switch between the nodes 
kern.cam.ctl.ha_role=0 # 0 is primary; 1 is secondary

# sysctl -a | grep ctl.ha
kern.cam.ctl.ha_peer: connect 172.16.1.3:7940
kern.cam.ctl.ha_role: 0
kern.cam.ctl.ha_link: 2
kern.cam.ctl.ha_id: 1
kern.cam.ctl.ha_mode: 2


Server 2 config (Secondary Node)
Code: 
# cat /boot/loader.conf
geom_mirror_load="YES"
geom_label_load="YES"
kern.geom.label.disk_ident.enable=0
hw.memtest.tests=0
geom_multipath_load="YES"
kern.cam.ctl.ha_id=2
kern.cam.ctl.ha_mode=2

# cat /etc/sysctl.conf
kern.cam.ctl.debug=2
kern.cam.ctl.ha_peer="listen 172.16.1.3:7940"
kern.cam.ctl.ha_role=1  # 0 is primary; 1 is secondary

# sysctl -a | grep ctl.ha
kern.cam.ctl.ha_peer: listen 172.16.1.3:7940
kern.cam.ctl.ha_role: 1
kern.cam.ctl.ha_link: 2
kern.cam.ctl.ha_id: 2
kern.cam.ctl.ha_mode: 2
So its connection is established in active/active proxy mode
( netstat -a on both nodes also shows established connections)

Server 1
ctladm(8) shows the HA ports
Code: 
# ctladm portlist
Port Online Frontend Name     pp vp
0    YES    tpc      tpc      0  0
1    NO     camsim   camsim   0  0  naa.5000000acfe0eb02
2    YES    ioctl    ioctl    0  0
3    YES    camtgt   isp0     0  0  naa.2100000e1ecb9ed0
4    YES    camtgt   isp1     0  0  naa.2100000e1ecb9ed1
5    YES    camtgt   isp2     0  0  naa.2100000e1ecbafc0
6    YES    camtgt   isp3     0  0  naa.2100000e1ecbafc1
128  YES    ha       2:tpc    0  0
129  YES    ha       2:camsim 0  0  naa.5000000acfe0eb82
130  YES    ha       2:ioctl  0  0
131  YES    ha       2:isp0   0  0  naa.2100000e1ecb99b0
132  YES    ha       2:isp1   0  0  naa.2100000e1ecb99b1
133  YES    ha       2:isp2   0  0  naa.2100000e1ecbad40
134  YES    ha       2:isp3   0  0  naa.2100000e1ecbad41
ctladm(8) shows the initiators are logging into the ports on the other node
Code: 
# ctladm portlist -i
Port Online Frontend Name     pp vp
0    YES    tpc      tpc      0  0
1    NO     camsim   camsim   0  0  naa.5000000acfe0eb02
  Target: naa.5000000acfe0eb00
2    YES    ioctl    ioctl    0  0
3    YES    camtgt   isp0     0  0  naa.2100000e1ecb9ed0
  Target: naa.2000000e1ecbafc1
  Initiator 1: naa.10000090faa9eb37
4    YES    camtgt   isp1     0  0  naa.2100000e1ecb9ed1
  Target: naa.2000000e1ecbafc1
  Initiator 1: naa.10000090faa9e93b
5    YES    camtgt   isp2     0  0  naa.2100000e1ecbafc0
  Target: naa.2000000e1ecbafc1
  Initiator 1: naa.10000090faa9eb37
6    YES    camtgt   isp3     0  0  naa.2100000e1ecbafc1
  Target: naa.2000000e1ecbafc1
  Initiator 1: naa.10000090faa9e93b
128  YES    ha       2:tpc    0  0
129  YES    ha       2:camsim 0  0  naa.5000000acfe0eb82
  Target: naa.5000000acfe0eb00
130  YES    ha       2:ioctl  0  0
131  YES    ha       2:isp0   0  0  naa.2100000e1ecb99b0
  Target: naa.2000000e1ecb99b0
  Initiator 1: naa.10000090faa9eb37
132  YES    ha       2:isp1   0  0  naa.2100000e1ecb99b1
  Target: naa.2000000e1ecb99b1
  Initiator 1: naa.10000090faa9e93b
133  YES    ha       2:isp2   0  0  naa.2100000e1ecbad40
  Target: naa.2000000e1ecbad40
  Initiator 1: naa.10000090faa9eb37
134  YES    ha       2:isp3   0  0  naa.2100000e1ecbad41
  Target: naa.2000000e1ecbad41
  Initiator 1: naa.10000090faa9e93b

 # ctladm portlist -l
Port Online Frontend Name     pp vp
0    YES    tpc      tpc      0  0
  All LUNs mapped
1    NO     camsim   camsim   0  0  naa.5000000acfe0eb02
  All LUNs mapped
2    YES    ioctl    ioctl    0  0
  All LUNs mapped
3    YES    camtgt   isp0     0  0  naa.2100000e1ecb9ed0
  LUN 0: 0
  LUN 1: 1
  LUN 2: 2
  LUN 3: 3
4    YES    camtgt   isp1     0  0  naa.2100000e1ecb9ed1
  LUN 0: 0
  LUN 1: 1
  LUN 2: 2
  LUN 3: 3
5    YES    camtgt   isp2     0  0  naa.2100000e1ecbafc0
  LUN 0: 0
  LUN 1: 1
  LUN 2: 2
  LUN 3: 3
6    YES    camtgt   isp3     0  0  naa.2100000e1ecbafc1
  LUN 0: 0
  LUN 1: 1
  LUN 2: 2
  LUN 3: 3
128  YES    ha       2:tpc    0  0
  All LUNs mapped
129  YES    ha       2:camsim 0  0  naa.5000000acfe0eb82
  All LUNs mapped
130  YES    ha       2:ioctl  0  0
  All LUNs mapped
131  YES    ha       2:isp0   0  0  naa.2100000e1ecb99b0
  All LUNs mapped
132  YES    ha       2:isp1   0  0  naa.2100000e1ecb99b1
  All LUNs mapped
133  YES    ha       2:isp2   0  0  naa.2100000e1ecbad40
  All LUNs mapped
134  YES    ha       2:isp3   0  0  naa.2100000e1ecbad41
  All LUNs mapped

So everything looks good ... but when I unplug the SAN connections on the primary node server 1 what I expect to happen is that it will continue to function by its connections to the secondary server and data move across the cluster interconnect. What happens is that the client server loses all connectivity to the LUNs. Tried this with same results on my FreeBSD array that uses iSCSI as well.

Ideas?
 
Hello, have you find a way to resolve the issue? I'm going to test it on fresh 11 release.
I'm also trying to implement HA with my own simple solution but it must be checked on real hardware.
 
Yes the key is that you have to have the luns defined on both nodes in the ctl.conf ... which is I was not expecting given the output above showing all luns presented.

hopefully I will be building another cluster in the near future and will do a better job of documenting the build and post an updated how-to
 
I have playing with it for a while, and here is my small report on trying to use the new CTL HA in the BeaST storage system.

Talking seriously, I am a little puzzled, it looks like the new CTL HA is not as highly available as it is supposed to be. Or I have configured it wrong :)
 
Forget my words :) CTL HA can do failover when assigning the primary role to the survived controller: sysctl kern.cam.ctl.ha_role=0 So I'm already updating my report. The new version is here.
 
You must log in or register to reply here.
Back
Top