ZFS help to write script to make ZFS backup

hello, I'am making a little script to make ZFS snapshots, send and receive,etc
so far so good :


Bash:
#!/usr/local/bin/bash

header="/files@backup"
fecha=`date -j +"%Y-%d-%m--%H:%M:%S"`
sep="-"
snap=$header$sep$fecha
echo "Making snapshot of /zroot/files" $snap
zfs snapshot  zroot$snap
echo "-----------------------------------------" >> /home/juan/log-snapshots
echo $snap >> /home/juan/log-snapshots
echo "-----------------------------------------" >> /home/juan/log-snapshots
zfs send zroot$snap | pv | ssh root@10.1.80.2 zfs receive -dvF zroot
zfs destroy  zroot$snap

the backup will be the zroot/files plus the date (looks very good and usefull)
now, how I'can know if zfs send returns without error?
after this
Code:
zfs send zroot$snap | pv | ssh root@10.1.80.2 zfs receive -dvF zroot

note:in the future I'will change to ssh keys metod to make it more automatized
the backup machine is my home and only do that, I'plug in it,make the backups and shutdown
 
Regarding checking the status of the pipe commands, if you are ok with using zsh instead of bash, I have the following example for checking if all commands in a pipe finished successfully:

Bash:
#!/usr/local/bin/zsh

zsh_checkPipeStatus()
{
    # https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another
    local psString=$pipestatus
    
    log_trace "$@"
    log_trace "psString=$psString"
    
    # https://unix.stackexchange.com/questions/266945/how-to-split-a-string-on-a-control-character
    # split string
    local psArray=(${(s: :)psString})
    
    for ps in $psArray; do
        log_trace "  >$ps< "
        [ $ps -ne 0 ] &&
        {
            log_trace "rtn $ps"
            return $ps
        }
    done
    
    log_trace "rtn 0"
    return 0
}

ls | grep abc | sort
zsh_checkPipeStatus || die "Pipe commands failed."

bash has a similar mechanism but I have not used it.
 
Regarding checking the status of the pipe commands, if you are ok with using zsh instead of bash, I have the following example for checking if all commands in a pipe finished successfully:

Bash:
#!/usr/local/bin/zsh

zsh_checkPipeStatus()
{
    # https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another
    local psString=$pipestatus
   
    log_trace "$@"
    log_trace "psString=$psString"
   
    # https://unix.stackexchange.com/questions/266945/how-to-split-a-string-on-a-control-character
    # split string
    local psArray=(${(s: :)psString})
   
    for ps in $psArray; do
        log_trace "  >$ps< "
        [ $ps -ne 0 ] &&
        {
            log_trace "rtn $ps"
            return $ps
        }
    done
   
    log_trace "rtn 0"
    return 0
}

ls | grep abc | sort
zsh_checkPipeStatus || die "Pipe commands failed."

bash has a similar mechanism but I have not used it.

wow,very elaborated, and clean, I'have to search more but I'been testing with the exit status of bash
Bash:
$?

so far so good, but in that line are 3 commands,the 2 test I'made were:

1: SSH error (wrong password) the
Bash:
$?
return a 255 code
2: ZFS send , really and error in receive
Code:
cannot receive new filesystem stream: destination has snapshots
return a 1 code
3:ZFS send and receive successful return a code 0

I'know about the exit status but not use for a long time..according to this page the above are the expecteds return code's
(except SSH,but is normal too)

Advanced Bash-Scripting Guide:

so,it seems reliable (checking the return code in every command)

I also wrote my own similar scripts until I learned about sysutils/zfsnap.

It's seems to be a good tool, for now i prefer make the scripts for almost everything
 
PS, or use "logger" instead of user logfile.

PS seems more complicated,not for the script write(using awk,sed,etc) but because the secuential order of the commands
(hard to take out of my head..sorry 😄) , and logger, yes, maybe in a future when I'made a more "professional" script
thanks for the tips
 
wow,very elaborated, and clean, I'have to search more but I'been testing with the exit status of bash
Bash:
$?

so far so good, but in that line are 3 commands,the 2 test I'made were:

1: SSH error (wrong password) the
Bash:
$?
return a 255 code
2: ZFS send , really and error in receive
Code:
cannot receive new filesystem stream: destination has snapshots
return a 1 code
3:ZFS send and receive successful return a code 0

I'know about the exit status but not use for a long time..according to this page the above are the expecteds return code's
(except SSH,but is normal too)

Advanced Bash-Scripting Guide:

so,it seems reliable (checking the return code in every command)

I also wrote my own similar scripts until I learned about sysutils/zfsnap.

It's seems to be a good tool, for now i prefer make the scripts for almost everything
The thing is, when you pipe commands, if a command inside the pipe fails you don't get the return code. You get only the return code of the last (rightmost) command in the pipe. Here is an example:
Code:
$ ls /gaga
$ echo $?
2
$ ls /gaga | sort
$ echo $?
0

You check the return code and it's 0 and you'll never know that ls /gaga failed unless you check the whole pipe commands' status as per my example.
 
The thing is, when you pipe commands, if a command inside the pipe fails you don't get the return code. You get only the return code of the last (rightmost) command in the pipe

Exactly , unless I'make a function between pipe's that include the command and the return code or something like
your script but in bash..work in progress :)
 
IMHO the general way to go is a combination of LUA ZFS channel programs (zfs-program(1)), driven by either
  1. Existing solution's wound neck is their dependance on external languages from the ports, mostly python2.7(1) or ruby(1) -- but such vital thing should not depend on ports:
    1.1 you may need to run it from a USB stick or /rescue environment (no shared libs!), and 2nd
    1.2 this is troublesome when e.g. the port changes it's default behaviour on an upgrade
    or else s/th brakes on a port-upgrade
  2. shell scripts are not maintainable once they reach a certain size, and they are error-prone.
    This is a commonly accepted "wisdom", e.g. look at the history of portmaster(8)
  3. LUA is better suited for modular programming than shell scripts and less error-prone.
  4. The best solution I know inspects a ZFS user property ("zfs-auto-snapshot") to decide if a dataset should be snapshot'ed. This avoids all trouble with exclude/include patterns (swap on ZFS, /var/{cache,tmp},...)
  5. This tool knows how to suspend/resume DB's before/after taking the snapshot.
Good luck! & KISS, but beware the pitfalls...
 
This seems like a big lift. I don't see any ability to use the ZFS Lua API in cli.lua(8). The zfs-program(8) man page is in section 8, not section 1, BTW.
When I read the man page, it shows itsself in section 1M, but this can't be used in the bbtags.
To use cli.lua(8) is indeed not what I intended... I just did not read it's purpose, I assumed it is for making a CLI (terminal command line interface) program. I guess there must be a way to create a CLI app with LUA.
Besides that, as I understand it, you can not use anything from cli.lua(8) from within a ZFS channel program, but you can drive a ZFS channel program from a lua.cli(8) program.
 
Bash:
zfs send zroot$snap | pv | ssh root@10.1.80.2 zfs receive -dvF zroot
now, how I'can know if zfs send returns without error?

I did not yet find a need to know that. If zfs send gets an error, the transfer will fail, so zfs recv will also get an error, and that one is returned in $?.
But:
  1. I do not know what the pv command does here - that doesn't exist on my system.
  2. I NEVER code anything other than #!/bin/sh, because that would lead straightways into hell's kitchen.
Now then, if you really need to grab the exitval of some earlier command in a pipe, that does require some tricks.
In a simple way you can route it via another handle until you get stdout free to read it:

Code:
$ cat nonexist 
cat: nonexist: No such file or directory
$ echo $?
1
$ cat nonexist | cat > /dev/null
cat: nonexist: No such file or directory
$ echo $?
0
So this is bad.
But this works:
$ ( (cat nonexist ; echo $? >&2 ) | cat > /dev/null ) 2>&1
If you need the error handle for other things, you can also use another one:
$ ( (cat nonexist ; echo $? >&3 ) | cat > /dev/null ) 3>&1
You only need to make sure that at the end of the pipe, you first dispose off all the things that might be on stdout, and get that free, before routing the errorcode back from the overdrive into stdout. Then You can grab it into a variable:
Code:
$ RC=`( (cat nonexist ; echo $? >&3 ) | cat  > /dev/null ) 3>&1`
cat: nonexist: No such file or directory
$ echo $?       # second cat
0
$ echo $RC      # first cat
1
 
Ah, that pv seems to be kind of a progress monitor...

That's then a bit different from my use-case. I have such thing run every 5 minutes from cron, and I do NOT want to see anything of it; it should be self-healing and self-maintaining and just keep the mirror current.

But then, that tends to get elaborate. So if you find a ready-made tool that suits your purpose, then I would generally recommend to use that.

Here is a glimpse of what I ended up with, so you can see what you're up to when coding such:

Code:
    # do the replication
    if test $full = full; then
        su - $Ruser -c "ssh $Rhost sudo /sbin/zfs destroy $recurse $targ@%" \
                2> /dev/null

        # when not recursive and other snapshots exist, full must be split
        if test ! $recurse; then
            first="`zfs list -H -d 1 -t snapshot -o name "$path" | head -1`"
        fi
        if ! if test -z "$first" -o "$first" = $path@$Snapp.$snapn; then
                zfs send ${recurse:+-R} -PLcepv $path@$Snapp.$snapn 2> $Tmpf.0|\
                    su - $Ruser -c "ssh $Rhost sudo /sbin/zfs recv -Fu $targ" \
                    2> /dev/null
            else
                zfs send -PLcepv $first 2> $Tmpf.0 | \
                    su - $Ruser -c "ssh $Rhost sudo /sbin/zfs recv -Fu $targ" \
                    2> /dev/null \
                && zfs send -PLcepv -I $first $path@$Snapp.$snapn 2>> $Tmpf.0 |\
                    su - $Ruser -c "ssh $Rhost sudo /sbin/zfs recv -Fu $targ" \
                    2> /dev/null
            fi; then
            zfs destroy $recurse $path@$Snapp.$snapn
            echo "FAIL-REM-1"
            return 1
        fi
    else
        if ! zfs send ${recurse:+-R} -PLcepv -I $path@$Snapp.$last \
             $path@$Snapp.$snapn 2> $Tmpf.0 | \
                su - $Ruser -c "ssh $Rhost sudo /sbin/zfs recv -Fu $targ" \
                2> /dev/null; then
            zfs destroy $recurse $path@$Snapp.$snapn
            echo "FAIL-REM-2"
            return 1        
        else
            # delete our old snapshots
            zfs list -H ${recurse:--d 1} -t snapshot -o name $path \
                    | grep "@$Snapp\." | while read snap; do
                if ! echo "$snap" | grep -q "@$Snapp.$snapn\$"; then
                    zfs destroy "$snap"
                fi
            done
            # when not recursive, delete (all!) remote snapshots not local
            if test ! "$recurse"; then
                su - $Ruser -c "ssh $Rhost /sbin/zfs list -H -d 1 \
                        -t snapshot -o name $targ" | while read snap; do
                        _l=`echo "$snap" | sed "s/^.*@//"`
                        if ! zfs list $path@$_l > /dev/null 2> /dev/null; then
                            su - $Ruser -c \
                               "ssh $Rhost sudo /sbin/zfs destroy $snap" \
                               < /dev/null
                        fi
                    done
            fi
        fi
    fi
 
Back
Top