Failover redundancy with CARP for two web servers in jails

B

Bogdan

I'm having trouble trying to setup carp(4) to provide fail-over redundancy for two web servers running in jails.

guest1: 192.168.255.100
/etc/rc.conf:
Code: 
ifconfig_re0_alias0="vhid 1 pass secret alias 192.168.255.140 netmask  255.255.255.255"

ifconfig
Code: 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   inet 192.168.255.100 netmask 0xffffff00 broadcast 192.168.255.255
   carp: MASTER vhid 1 advbase 1 advskew 100
   carp: MASTER vhid 2 advbase 1 advskew 100


guest2: 192.168.255.101
/etc/rc.conf
Code: 
ifconfig_re0_alias0="vhid 2 pass secret alias 192.168.255.141 netmask 255.255.255.255"

ifconfig
Code: 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   inet 192.168.255.101 netmask 0xffffff00 broadcast 192.168.255.255
   carp: MASTER vhid 1 advbase 1 advskew 100
   carp: MASTER vhid 2 advbase 1 advskew 100

host: 192.168.255.226
/etc/rc.conf
Code: 
ifconfig_re0="inet 192.168.255.226 netmask 255.255.255.0"

#Network interfaces for jailed nginx
ifconfig_re0_alias3="inet 192.168.255.100 netmask 255.255.255.0"
ifconfig_re0_alias4="inet 192.168.255.101 netmask 255.255.255.0"

ifconfig_re0_alias5="inet vhid 1 advskew 100 pass secret alias 192.168.255.140 netmask 255.255.255.255"
ifconfig_re0_alias6="inet vhid 2 advskew 100 pass secret alias 192.168.255.141 netmask 255.255.255.255"

ifconfig
Code: 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
   inet 192.168.255.226 netmask 0xffffff00 broadcast 192.168.255.255
   inet 192.168.255.100 netmask 0xffffff00 broadcast 192.168.255.255
   inet 192.168.255.101 netmask 0xffffff00 broadcast 192.168.255.255
   inet 192.168.255.140 netmask 0xffffffff broadcast 192.168.255.140 vhid 1
   inet 192.168.255.141 netmask 0xffffffff broadcast 192.168.255.141 vhid 2
   carp: MASTER vhid 1 advbase 1 advskew 100
   carp: MASTER vhid 2 advbase 1 advskew 100

I cannot access the servers via 192.168.255.226 (re0).

I followed the tutorial from the Handbook.

I am also not sure the rc.conf files in the jails should have the interface aliases there, since those interfaces are not configured via local rc.conf files, but on the host (/etc/rc.conf and /usr/local/etc/ezjail/www_localdomain).

I'm definitely doing something the wrong way, but I can't figure out what. Any suggestions would be much appreciated!
 
Fail-over redundancy isn't going to do you much good if both are running on the same system. If the host dies both your webservers will be offline. carp(4) is meant to be run on two different hosts, I'm not even sure if it's going to work at all between two jails on the same host.
 
If you want to have two instances of the webserver for upgrading purposes, you could play around with pf and a script, which switches over the traffic between the internal IPs.
So you can upgrade the "offline" instance while serving with the other one. The public IP would be handled by the host.

Regards
Markus
 
Another option would be to install net/haproxy on the host and use that to load-balance and have it fail-over. That way the fail-over will work pretty much automatically and there would be no need for NAT redirection. It will also allow you to direct different sites to different jails based on the URL.
 
Thank you, SirDice and storvi_net, I appreciate very much your feedback and given suggestions.

Indeed, two servers on the same machine is not a good idea. I'm doing this just to learn more and I only have one machine to play with.

I could have two instances of FreeBSD with VirtualBox and probably everything would be fine with carp() and failover redundancy. But after spending so many hours, trying to make it work with servers in jails, I feel like I would be cheating on myself... taking the easy path in life. :))

I heard about net/haproxy and it's definitely an application I want to try, but I want to play a little bit more with carp(), maybe there is a solution.

Meanwhile, I realised it's not necessary to alter the /etc/rc.conf files on guests. The following lines on host are enough:

/usr/local/etc/ezjail/www_localdomain
Code: 
export jail_www_localdomain_ip="192.168.255.100,192.168.255.140"

/usr/local/etc/ezjail/www2_localdomain
Code: 
export jail_www2_localdomain_ip="192.168.255.101,192.168.255.141"

/etc/rc.conf
Code: 
ifconfig_re0_alias5="inet vhid 1 advskew 100 pass secret alias 192.168.255.140 netmask 255.255.255.255"
ifconfig_re0_alias6="inet vhid 2 advskew 100 pass secret alias 192.168.255.141 netmask 255.255.255.255"

Now both .100 and .140 interfaces show up with ifconfig on guests (I also replaced the /24 netmask with /32 for every single network address exported to any jails in the /etc/rc.conf file on host:

Code: 
re0:
  inet 192.168.255.100 netmask 0xffffffff broadcast 192.168.255.100
  inet 192.168.255.140 netmask 0xffffffff broadcast 192.168.255.140 vhid 1
  status: active
  carp: MASTER vhid 1 advbase 1 advskew 100
  carp: MASTER vhid 2 advbase 1 advskew 100

Here I am now:

dmesg -a | grep carp
Code: 
carp: BACKUP vhid 1 advbase 1 advskew 100
carp: BACKUP vhid 2 advbase 1 advskew 100
carp: VHID 2@re0: BACKUP -> MASTER (master down)
carp: VHID 1@re0: BACKUP -> MASTER (master down)
carp: demoted by 240 to 240 (send error 13 on re0)
carp: demoted by 240 to 480 (send error 13 on re0)

and in the ipfw log:

Code: 
Oct  7 17:58:57 host kernel: ipfw: 299 Deny P:112 192.168.255.226 224.0.0.18 out via re0

I need to check out the firewall rules, too, and I'll post updates in case of any progress.
 
Finished with installing and running net/haproxy. It provides both failover redundancy and loadbalancing for both jailed nginx servers.

Happy with the config!
 
Bogdan said:
Thank you, SirDice and storvi_net, I appreciate very much your feedback and given suggestions.

Indeed, two servers on the same machine is not a good idea. I'm doing this just to learn more and I only have one machine to play with.

I could have two instances of FreeBSD with VirtualBox and probably everything would be fine with carp() and failover redundancy. But after spending so many hours, trying to make it work with servers in jails, I feel like I would be cheating on myself... taking the easy path in life. :))

I heard about net/haproxy and it's definitely an application I want to try, but I want to play a little bit more with carp(), maybe there is a solution.

Meanwhile, I realised it's not necessary to alter the /etc/rc.conf files on guests. The following lines on host are enough:

/usr/local/etc/ezjail/www_localdomain
Code: 
export jail_www_localdomain_ip="192.168.255.100,192.168.255.140"

/usr/local/etc/ezjail/www2_localdomain
Code: 
export jail_www2_localdomain_ip="192.168.255.101,192.168.255.141"

/etc/rc.conf
Code: 
ifconfig_re0_alias5="inet vhid 1 advskew 100 pass secret alias 192.168.255.140 netmask 255.255.255.255"
ifconfig_re0_alias6="inet vhid 2 advskew 100 pass secret alias 192.168.255.141 netmask 255.255.255.255"

Now both .100 and .140 interfaces show up with ifconfig on guests (I also replaced the /24 netmask with /32 for every single network address exported to any jails in the /etc/rc.conf file on host:

Code: 
re0:
  inet 192.168.255.100 netmask 0xffffffff broadcast 192.168.255.100
  inet 192.168.255.140 netmask 0xffffffff broadcast 192.168.255.140 vhid 1
  status: active
  carp: MASTER vhid 1 advbase 1 advskew 100
  carp: MASTER vhid 2 advbase 1 advskew 100

Here I am now:

dmesg -a | grep carp
Code: 
carp: BACKUP vhid 1 advbase 1 advskew 100
carp: BACKUP vhid 2 advbase 1 advskew 100
carp: VHID 2@re0: BACKUP -> MASTER (master down)
carp: VHID 1@re0: BACKUP -> MASTER (master down)
carp: demoted by 240 to 240 (send error 13 on re0)
carp: demoted by 240 to 480 (send error 13 on re0)

and in the ipfw log:

Code: 
Oct  7 17:58:57 host kernel: ipfw: 299 Deny P:112 192.168.255.226 224.0.0.18 out via re0

I need to check out the firewall rules, too, and I'll post updates in case of any progress.
Click to expand...

I know this is an old thread but I think the reason this never worked for you is because CARP requires full access to the NIC so your jails would have had to be VNET jails for this to actually work.
 
You must log in or register to reply here.
Back
Top