Solved Carp-Fail-Over

fr33bsd

fr33bsd

Hi people,

for a test setup of 2 vms

vm1:
Code: 
hostname="hosta.example.org"
ifconfig_em0="inet 192.168.1.3 netmask 255.255.255.0"
ifconfig_em0_alias0="inet vhid 1 pass testpass alias 192.168.1.100/32"

vm2:
Code: 
hostname="hostb.example.org"
ifconfig_em0="inet 192.168.1.4 netmask 255.255.255.0"
ifconfig_em0_alias0="inet vhid 1 advskew 100 pass testpass alias 192.168.1.100/32"

I would like to write a fail-over-script for hast.

Before you scream, yes, I know, there are already some scripts like this one for ucarp or that little older one from 2012.

I cannot use the ucarp one, because in my setup I use carp.
The little older one from 2012 could be used, but there is this tiny difference in the network setup cloned_interfaces="carp0", so I cannot actually use
Code: 
notify 30 {
    match "system" "IFNET";
    match "subsystem" "carp0";
    match "type" "LINK_UP";
    action "/usr/local/bin/failover master";
};

notify 30 {
    match "system" "IFNET";
    match "subsystem" "carp0";
    match "type" "LINK_DOWN";
    action "/usr/local/bin/failover slave";
};
in /etc/devd.conf on both nodes to trigger the fail-over script.

And in general, of course you should setup a dedicated nic acting as wire backbone ? for ha-heartbeats and data sync of a ha-storage if bandwidth is low. But this is not the case here.

Later I want to expand the setup little up to 2 nics per vms, maybe using lagg (failover mode), so the carp device thing in /etc/devd.conf might not work anymore, because none of both nics will go down if - for example - one switch dies and nic1 will be unavailable on each machine and lagg nic dummy device will stay online.

So triggering the failover script does not rely on link status ...

At this moment, I am actually out-of-ideas how to handle it. Got lost for today. :rolleyes:

I appreciate each help.
 
Last edited:
I'm using this:
Code: 
notify 0 {
        match "system"    "CARP";
        match "subsystem" "[0-9]+@[0-9a-z]+";
        match "type"      "(MASTER|BACKUP)";
        action "/root/bin/carpcontrol.sh $type $subsystem";
};
Saved as /etc/devd/carp.conf.
The start of that carpcontrol.sh script looks like this:
Code: 
#!/bin/sh

TYPE=$1
SUBSYS=$2

logger State change on ${SUBSYS} to ${TYPE}

case $TYPE in

        MASTER)         STATE=1 ;;
        BACKUP)         STATE=2 ;;
        *)              STATE=4 ;;

esac
 
SirDice said:
I'm using this:
Code: 
notify 0 {
        match "system"    "CARP";
        match "subsystem" "[0-9]+@[0-9a-z]+";
        match "type"      "(MASTER|BACKUP)";
        action "/root/bin/carpcontrol.sh $type $subsystem";
};
Click to expand...
So, devd senses if the status of carp has changed and it is launching the script with the specific parameters regarding the status, right?

Please, tell me in few words, how does it work (I do not want to read the source code today)? I mean, does devd interact with carp via internal mechanisms?
 
devd(8) interacts with devices being added, removed or changing state. In this case I told it to watch for state changes in CARP, if it detects them it'll run the 'action' script with the $type and $subsystem (that's the actual interface that had the state change; em0 for example) as arguments. That way I can use that information in my script. Depending on the outcome of the $STATE variable I do certain things, like starting or stopping a service.

Code: 
     The devd utility is a system daemon that runs in the background all the
     time.  Whenever a device is added to or removed from the device tree,
     devd will execute actions specified in devd.conf(5).
 
SirDice said:
devd(8) interacts with devices being added, removed or changing state. In this case I told it to watch for state changes in CARP, if it detects them it'll run the 'action' script with the $type and $subsystem (that's the actual interface that had the state change; em0 for example) as arguments. That way I can use that information in my script.

Code: 
     The devd utility is a system daemon that runs in the background all the
     time.  Whenever a device is added to or removed from the device tree,
     devd will execute actions specified in devd.conf(5).
Click to expand...
Now I understand, why using cron in this thread is non-sense. ? At least, non-sense in a way, because actions there had to be triggered by load and by number of (not-)running processes. :-/ How ever, let me try to focus on the failover script... :-/
 
SirDice said:
I'm using this:
Code: 
notify 0 {
        match "system"    "CARP";
        match "subsystem" "[0-9]+@[0-9a-z]+";
        match "type"      "(MASTER|BACKUP)";
        action "/root/bin/carpcontrol.sh $type $subsystem";
};
Click to expand...
So, the subsystem is one of the carp device, because there can be more than one carp configuration on one system, right?

In my case, case of only 1 carp device, I could go less generic and ignore subsystem as parameter for my carpcontrol script, but not in the devd.conf, right?

Something like this?

Code: 
#!/bin/sh
TYPE=$1
logger State change to ${TYPE}

# Original script by Freddie Cash <fjwcash@gmail.com>
# Modified by Michael W. Lucas <mwlucas@BlackHelicopters.org>
# and Viktor Petersson <vpetersson@wireload.net>
# Modified by George Kontostanos <gkontos.mail@gmail.com>

# The names of the HAST resources, as listed in /etc/hast.conf
resources="disk1 disk2 disk3"

# delay in mounting HAST resource after becoming master
# make your best guess
delay=3

# logging
log="local0.debug"
name="failover"
pool="zhast"

# end of user configurable stuff

case $TYPE in
    MASTER)
        logger -p $log -t $name "Switching to primary provider for ${resources}."
        sleep ${delay}

        # Wait for any "hastd secondary" processes to stop
        for disk in ${resources}; do
            while $( pgrep -lf "hastd: ${disk} \(secondary\)" > /dev/null 2>&1 ); do
                sleep 1
            done

            # Switch role for each disk
            hastctl role primary ${disk}
            if [ $? -ne 0 ]; then
                logger -p $log -t $name "Unable to change role to primary for resource ${disk}."
                exit 1
            fi
        done

        # Wait for the /dev/hast/* devices to appear
        for disk in ${resources}; do
            for I in $( jot 60 ); do
                [ -c "/dev/hast/${disk}" ] && break
                sleep 0.5
            done

            if [ ! -c "/dev/hast/${disk}" ]; then
                logger -p $log -t $name "GEOM provider /dev/hast/${disk} did not appear."
                exit 1
            fi
        done

        logger -p $log -t $name "Role for HAST resources ${resources} switched to primary."


        logger -p $log -t $name "Importing Pool"
        # Import ZFS pool. Do it forcibly as it remembers hostid of
                # the other cluster node.
                out=`zpool import -f "${pool}" 2>&1`
                if [ $? -ne 0 ]; then
                    logger -p local0.error -t hast "ZFS pool import for resource ${resource} failed: ${out}."
                    exit 1
                fi
                logger -p local0.debug -t hast "ZFS pool for resource ${resource} imported."

    ;;

    BACKUP)
        logger -p $log -t $name "Switching to secondary provider for ${resources}."

        # Switch roles for the HAST resources
        zpool list | egrep -q "^${pool} "
            if [ $? -eq 0 ]; then
                    # Forcibly export file pool.
                    out=`zpool export -f "${pool}" 2>&1`
                        if [ $? -ne 0 ]; then
                            logger -p local0.error -t hast "Unable to export pool for resource ${resource}: ${out}."
                            exit 1
                     fi
                    logger -p local0.debug -t hast "ZFS pool for resource ${resource} exported."
            fi
        for disk in ${resources}; do
            sleep $delay
            hastctl role secondary ${disk} 2>&1
            if [ $? -ne 0 ]; then
                logger -p $log -t $name "Unable to switch role to secondary for resource ${disk}."
                exit 1
            fi
            logger -p $log -t $name "Role switched to secondary for resource ${disk}."
        done
    ;;
esac

This should be it. Time to test and find out ... :-/ ;)
 
You must log in or register to reply here.
Back
Top