ZFS - Auto Snapshots

Leander · Sep 11, 2014

This may help you if you're looking for a snapshot management tool in order to have

automatic creation of ZFS snapshots
for your SAMBA vfs_shadow_copy2
of your choice of availabel ZFS on your system
or all available ZFS on your system
keep [n] snaphots of hourly,daily,weekly,monthly,yearly

The script provided below can be invoked either as

/usr/local/bin/ZFS-Snapshot [hourly|daily|weekly|monthly|yearly] [Keep n Snapshots]
OR /usr/local/bin/ZFS-Snapshot [hourly|daily|weekly|monthly|yearly] [Keep n Snapshots] 'Tank/FS1 Tank/FS2 [etc.]

eg.: /usr/local/bin/ZFS-Snapshot hourly 24 'Tank/FS1 Tank/FS2' or /usr/local/bin/ZFS-Snapshot monthly 3
This gives you the freedom of either preselect pools which you want to create managed snapshots or leave blank in order to create snapshots of ALL available ZFS tanks on your system.

The code below can simply be copied and pasted into your shell (if you use bash - otherwhise be careful!!!) - Also be careful if you copy and paste, cause those lines

Code:

while read -r NAME USED AVAIL REFER MOUNTPOINT; do
    if [ ! -z "$( echo "${NAME}" | grep -E '/')" ]; then
        zfs destroy "${NAME}"
    fi
done <<< "$(zfs list -t snapshot)"

will destroy ANY and ALL presious snapshots.

!!! BEFORE YOU USE IT:

IT'S NOT A BUG - IT'S A FEATURE: It will not touch ANY of your previous or ANY of your individually/manually created snapshots - so coexistence of manual/individual snapshots next to the managed once of this script is possible. The use is questionable though but may be useful to some of you.
in order to keep trancparancy I recomment to destroy all previous snapshots you've created thus far. BUT you don't have to
This script mgmt. tool stores its snapshot history in /var/db/ZFS-Snapshots/{Hourly,Daily,Weekly,Monthly,Yearly} - this means it won't be available for exported ZFS by default - you would have to backup this data manually

#
# ==================================================================
#

==> I recooment to copy and paste the following code into your local texteditor and quickly browse through it before you use it in any way ... because it deletes previous snapshots and sets up cronjobs ...

Code:

(
cat <<'EOF'
#!/usr/local/bin/bash
#
# Manage ZFS Snapshots
#
# zfs list -t snapshot
#
# ====================================================================================== #


KEEP="${2}"
OF_ZFS="${3}"
DB_ROOT="/var/db/ZFS-Snapshots"

#
# ====================================================================================== #
#

# Ensure, that envirnoment is setup
[ ! -d "${DB_ROOT}" ] && mkdir -p "${DB_ROOT}"
chmod -R 0755       "${DB_ROOT}"
chown -R root:wheel "${DB_ROOT}"


cd "${DB_ROOT}"
for DB_FILE in Hourly Daily Weekly Monthly Yearly; do
    [ ! -f "${DB_FILE}" ] && touch "${DB_FILE}"
    chmod 0755       "${DB_FILE}"
    chown root:wheel "${DB_FILE}"
done
cd ~


create_snapshot () {
    local OF_ZFS="${1}"
    local SNAPSHOT_NAME="$(TZ=GMT date +GMT-%Y-%m-%d_%H-%M-%S)"
    # Take a snapshot of all ${OF_ZFS}
    if [ ! -z "${OF_ZFS}" ]; then
        for ZFS in ${OF_ZFS}; do
            while read -r NAME USED AVAIL REFER MOUNTPOINT; do
                if [ ! -z "$( echo "${NAME}" | grep -E '/')" ] && [ "$( echo "${NAME}" | grep "${ZFS}")" ]; then
                        #echo "Snapshot of ${NAME}"
                        zfs snapshot ${NAME}@${SNAPSHOT_NAME}
                        echo "${NAME}@${SNAPSHOT_NAME}"
                fi
            done <<< "$( zfs list -H )"
        done

    # Take a snapshot of all ZFS, since ${OF_ZFS} is empty / not set
    else
        while read -r NAME USED AVAIL REFER MOUNTPOINT; do
            if [ ! -z "$( echo "${NAME}" | grep -E '/')" ]; then
                    #echo "Snapshot of ${NAME}"
                    zfs snapshot ${NAME}@${SNAPSHOT_NAME}
                    echo "${NAME}@${SNAPSHOT_NAME}"
            fi
        done <<< "$( zfs list -H )"
    fi
}



delete_from_DB () {
    local SNAPSHOT="${1}"
    local DB_FILE="${2}"

    # Escape Slashes
    ESCAPED_SNAPSHOT="$( echo "${SNAPSHOT}" | sed -e 's|\/|\\/|' )"

    # Remove from DB
    RESULT="$( cat "${DB_FILE}" | sed "/${ESCAPED_SNAPSHOT}/d" )"
    echo "${RESULT}" > "${DB_FILE}"
}



destroy_Snapshots () {

    local KEEP="${1}"
    local DB_FILE="${2}"

    # Separate ZFS children
    while read -r LINE; do
        if [ ! -z "${LINE}" ]; then
            # Get all Snapshots of ZFS child
            COUNT=0
            while read -r SNAPSHOT; do
                if [ ${COUNT} -ge ${KEEP} ]; then
               #    echo "Destroy ${SNAPSHOT}"
                    zfs destroy ${SNAPSHOT}
                    delete_from_DB "${SNAPSHOT}" "${DB_FILE}"
               #else
               #    echo "Keep ${SNAPSHOT}"
                fi
                COUNT=$(( ${COUNT} + 1 ))
            done <<< "$( cat "${DB_FILE}" | grep "${LINE}" | sort -r )"
        fi
    done <<< "$( cat "${DB_FILE}" | sed -E 's|(.*)@.*|\1|' | sort -r --unique )"
}


# ====================================================================================== #


# Create Snapshots every Hour and keep at least ${KEEP_HOURS} Snapshots
if [ ! -z "$(echo "${1}" | grep -Ei 'hourly' )" ]; then
    DB_FILE="${DB_ROOT}/Hourly"
    create_snapshot "${OF_ZFS}" >> "${DB_FILE}"
    destroy_Snapshots "${KEEP}" "${DB_FILE}"
fi


# Create Snapshots every Day and keep at least ${KEEP_DAYS} Snapshots
if [ ! -z "$(echo "${1}" | grep -Ei 'daily' )" ]; then
    DB_FILE="${DB_ROOT}/Daily"
    create_snapshot "${OF_ZFS}" >> "${DB_FILE}"
    destroy_Snapshots "${KEEP}" "${DB_FILE}"
fi


# Create Snapshots every Week and keep at least ${KEEP_WEEKS} Snapshots
if [ ! -z "$(echo "${1}" | grep -Ei 'weekly' )" ]; then
    DB_FILE="${DB_ROOT}/Weekly"
    create_snapshot "${OF_ZFS}" >> "${DB_FILE}"
    destroy_Snapshots "${KEEP}" "${DB_FILE}"
fi


# Create Snapshots every Month and keep at least ${KEEP_MONTHS} Snapshots
if [ ! -z "$(echo "${1}" | grep -Ei 'monthly' )" ]; then
    DB_FILE="${DB_ROOT}/Monthly"
    create_snapshot "${OF_ZFS}" >> "${DB_FILE}"
    destroy_Snapshots "${KEEP}" "${DB_FILE}"
fi


# Create Snapshots every Year and keep at least ${KEEP_YEARS} Snapshots
if [ ! -z "$(echo "${1}" | grep -Ei 'yearly' )" ]; then
    DB_FILE="${DB_ROOT}/Yearly"
    create_snapshot "${OF_ZFS}" >> "${DB_FILE}"
    destroy_Snapshots "${KEEP}" "${DB_FILE}"
fi


EOF
) >              /usr/local/bin/ZFS-Snapshot
chmod 0755       /usr/local/bin/ZFS-Snapshot
chown root:wheel /usr/local/bin/ZFS-Snapshot







#
# Delete all current Snapshots in order to prevent zombies outside of the DB setup below
#

# Delete all current snapshots
while read -r NAME USED AVAIL REFER MOUNTPOINT; do
    if [ ! -z "$( echo "${NAME}" | grep -E '/')" ]; then
        zfs destroy "${NAME}"
    fi
done <<< "$(zfs list -t snapshot)"





#
# Setup Snapshot Management DB
#

mkdir -p -m 0755    /var/db/ZFS-Snapshots

echo '' > /var/db/ZFS-Snapshots/Hourly  # Snapshots of the day
echo '' > /var/db/ZFS-Snapshots/Daily   # Snapshots of the day
echo '' > /var/db/ZFS-Snapshots/Weekly  # Snapshots of the week
echo '' > /var/db/ZFS-Snapshots/Monthly # Snapshots of the month
echo '' > /var/db/ZFS-Snapshots/Yearly  # Snapshots of the Year

chmod -R 0755       /var/db/ZFS-Snapshots
chown -R root:wheel /var/db/ZFS-Snapshots





#
# Setup CronJobs
#

# Clean CronTab
RESULT="$(crontab -l | sed '/.*ZFS.*Snapshot.*/d')"
new_CronJob="
${RESULT}
" && \
( echo "$new_CronJob") | crontab -


[[ -z "$( crontab -l 2>&1 | grep -E 'ZFS Snapshots' )" ]] && \
new_CronJob="
# ZFS Snapshots:
0    *    *    *    *    /usr/local/bin/ZFS-Snapshot hourly  24 'Tank/FS1 Tank/FS2'
0    0    *    *    *    /usr/local/bin/ZFS-Snapshot daily   7  'Tank/FS1 Tank/FS2'
0    0    *    *    0    /usr/local/bin/ZFS-Snapshot weekly  2  'Tank/FS1 Tank/FS2'
0    0    1    *    *    /usr/local/bin/ZFS-Snapshot monthly 2  'Tank/FS1 Tank/FS2'
0    0    1    1    *    /usr/local/bin/ZFS-Snapshot yearly  2  'Tank/FS1 Tank/FS2'
" && \
(crontab -l; echo "$new_CronJob") | crontab -

My correlating smb4.conf looks like:

Code:

[...]
  vfs objects = shadow_copy2, zfsacl, streams_xattr, recycle
  zfsacl:acesort = dontcare
  nfs4:mode      = special
  nfs4:acedup    = merge
  nfs4:chown     = yes
  shadow:format  = GMT-%Y-%m-%d_%H-%M-%S
  shadow:sort    = desc
  shadow:snapdir = .zfs/snapshot
[...]

Some infos on my ZFS config (relevant or not ... try and see what you find useful):

Code:

zfs set snapdir=hidden Tank
zfs set aclinherit=passthrough Tank/FS1
zfs set aclmode=passthrough Tank/FS1

Let me know if you find it useful or have something to optimize on it.

Thanks

Sebulon · Sep 11, 2014

Hi @Leander!

Not to diminish the work you´ve done, but how is this different to sysutils/zfsnap?

/Sebulon

Leander · Sep 11, 2014

Its a simple script - not a port

Sebulon · Sep 11, 2014

Leander said:
Its a simple script - not a port

Gotcha!

/Sebulon

Leander · Sep 11, 2014

ZFSnap is great to use if it serves your needs. Also this script & method is way more light weight than what zfsnap is. Here it is simply one script and the according cronjob - bam that's it

There is too many people out there trying to achieve the same as I tried with a simple script - and here is a bricket or even the solution for their problem. Feel free to use and optimize my work.

SeriousEE · Nov 26, 2014

First of all, thank you - it is exactly I was searching for. The second thing:
For testing purpose I used two filesystems:

Code:

store01/pTest  4.61M  488G  4.61M  /store01/pTest
store01/pTest2  35K  488G  35K  /store01/pTest2

After running this command two times: /usr/local/bin/bash ZFS-Snapshot.sh hourly 2 store01, I found these entries in /var/db/ZFS-Snapshots/Hourly:

Code:

store01/pTest@GMT-2014-11-25_15-40-26
store01/pTest@GMT-2014-11-25_15-40-45

I expected to find four entries (two lines each filesystem). The reason is this line in function destroy_Snapshots:

Code:

done <<< "$( cat "${DB_FILE}" | grep "${LINE}" | sort -r )"

Because of the nearly identical names of the file systems the grep command from "pTest" also finds the entries from "pTest2". My fix was to add a @ sign in the grep command: done <<< "$( cat "${DB_FILE}" | grep "${LINE}@" | sort -r )". Maybe it is an improvement, otherwise thanks for the script.

ZFS - Auto Snapshots

Leander

Sebulon

Leander

Sebulon

Leander

SeriousEE