ZFS automatic snapshots with Sanoid on FreeBSD

  • Thread starter Thread starter Mark Phillips
  • Start date Start date
M

Mark Phillips

Guest
OpenZFS is a first class citizen on FreeBSD for a number of years now. Right from the installer, users can install a complete root-on-ZFS system. FreeBSD will create the necessary datasets for the base operating system, including one for /usr, /var, and /home.


If disaster strikes and files get lost, ZFS can restore data by rolling back to a previous snapshot – given that one exists for that particular dataset. The missing piece is a functionality to take automatic snapshots in regular intervals and rotate them. Rotation means that older snapshots get removed based on a certain user-defined policy. This avoids filling the dataset with millions of snapshots over time. While OpenZFS provides all the basic commands to take and remove snapshots, the user is responsible for managing their own snapshot agenda. This means how often a snapshot should occur on each dataset and what the rules for their removal are.


Over the years, programmers created separate solutions that offered this automatic snapshot functionality. Available as third-party software via the FreeBSD ports collection or precompiled packages, many of them are quick and easy to set up. This article takes a closer look at sanoid, written by Jim Salter. It offers the user to write their own policy that determines when and how often snapshots should happen. For example, system administrators may chose to snapshot the home dataset more frequently than the /usr/src dataset. The former contains critical and frequently changing data, whereas the latter is easily re-created from scratch (unless you’re a developer working on code in there).


Sanoid’s configuration file sanoid.conf located in the /usr/local/etc defines the policy. Consider the following example policy:


Code: 
[template_zroot]
  hourly = 36
  daily = 14
  monthly = 6
  recursive = yes
  autosnap = yes
  autoprune = yes
  frequent_period = 15
  frequently = 2

Here, we instruct sanoid to take and retain 36 hourly snapshots. Once the 37th snapshot is taken, the oldest one is removed. We’re keeping 14 daily snapshots (two weeks) and half a year of monthly snapshots. The snapshots themselves are taken recursively, meaning datasets below the zroot (in this case) also get snapshots. This is true for the current and any future snapshots that do not exist yet. This ensures that no dataset gets missed by the policy. The autosnap = yes line tells sanoid to take these snapshots automatically and give them a custom name. With the autoprune option activated, sanoid will remove older snapshots automatically. The frequent_period setting defines how many snapshots to take under an hour. In this case, sanoid takes a new snapshot every 15 minutes (4 per hour). Similarly, the frequently parameter defines how many snapshots under the hour to keep when pruning happens. Here we keep two snapshots of each hour.


When datasets need a different snapshot frequency, define a separate policy for them. The following line separates policies (below) from the datasets they should apply to (above).


Code: 
#############################
# templates below this line #
#############################

We have created an Ansible playbook that sets up sanoid on a FreeBSD system and configures the above policy. Sanoid executes from /etc/crontab and the playbook takes an initial snapshot as the last step of the playbook. With this, you will always have a snapshot available to retrieve an older state of your datasets. Tweak the numbers in the policy to have fewer or even more snapshots, based on how frequent the data changes and how much pool space is available.


Take a look at syncoid (part of the sanoid package) by the same author to transfer these snapshots via zfs send to a separate system for backup purposes.


We’re dropping new posts — and videos — for technical topics on a regular basis. So make sure you’re subscribed to the YouTube channel and following this feed in your favourite RSS reader. There’s also the newsletter, if you’d like to receive updates by email.

We’d like this content to be interactive — so what would you like to see us cover? What FreeBSD related topics puzzling you could we help with? Feel free to get in touch with your ideas.


Contributed by Benedict Reuschling


The post ZFS automatic snapshots with Sanoid on FreeBSD first appeared on FreeBSD Foundation.

Continue reading...
 
Mark Phillips said:
OpenZFS is a first class citizen on FreeBSD for a number of years now. Right from the installer, users can install a complete root-on-ZFS system. FreeBSD will create the necessary datasets for the base operating system, including one for /usr, /var, and /home.


If disaster strikes and files get lost, ZFS can restore data by rolling back to a previous snapshot – given that one exists for that particular dataset. The missing piece is a functionality to take automatic snapshots in regular intervals and rotate them. Rotation means that older snapshots get removed based on a certain user-defined policy. This avoids filling the dataset with millions of snapshots over time. While OpenZFS provides all the basic commands to take and remove snapshots, the user is responsible for managing their own snapshot agenda. This means how often a snapshot should occur on each dataset and what the rules for their removal are.


Over the years, programmers created separate solutions that offered this automatic snapshot functionality. Available as third-party software via the FreeBSD ports collection or precompiled packages, many of them are quick and easy to set up. This article takes a closer look at sanoid, written by Jim Salter. It offers the user to write their own policy that determines when and how often snapshots should happen. For example, system administrators may chose to snapshot the home dataset more frequently than the /usr/src dataset. The former contains critical and frequently changing data, whereas the latter is easily re-created from scratch (unless you’re a developer working on code in there).


Sanoid’s configuration file sanoid.conf located in the /usr/local/etc defines the policy. Consider the following example policy:


Code: 
[template_zroot]
  hourly = 36
  daily = 14
  monthly = 6
  recursive = yes
  autosnap = yes
  autoprune = yes
  frequent_period = 15
  frequently = 2

Here, we instruct sanoid to take and retain 36 hourly snapshots. Once the 37th snapshot is taken, the oldest one is removed. We’re keeping 14 daily snapshots (two weeks) and half a year of monthly snapshots. The snapshots themselves are taken recursively, meaning datasets below the zroot (in this case) also get snapshots. This is true for the current and any future snapshots that do not exist yet. This ensures that no dataset gets missed by the policy. The autosnap = yes line tells sanoid to take these snapshots automatically and give them a custom name. With the autoprune option activated, sanoid will remove older snapshots automatically. The frequent_period setting defines how many snapshots to take under an hour. In this case, sanoid takes a new snapshot every 15 minutes (4 per hour). Similarly, the frequently parameter defines how many snapshots under the hour to keep when pruning happens. Here we keep two snapshots of each hour.


When datasets need a different snapshot frequency, define a separate policy for them. The following line separates policies (below) from the datasets they should apply to (above).


Code: 
#############################
# templates below this line #
#############################

We have created an Ansible playbook that sets up sanoid on a FreeBSD system and configures the above policy. Sanoid executes from /etc/crontab and the playbook takes an initial snapshot as the last step of the playbook. With this, you will always have a snapshot available to retrieve an older state of your datasets. Tweak the numbers in the policy to have fewer or even more snapshots, based on how frequent the data changes and how much pool space is available.


Take a look at syncoid (part of the sanoid package) by the same author to transfer these snapshots via zfs send to a separate system for backup purposes.


We’re dropping new posts — and videos — for technical topics on a regular basis. So make sure you’re subscribed to the YouTube channel and following this feed in your favourite RSS reader. There’s also the newsletter, if you’d like to receive updates by email.

We’d like this content to be interactive — so what would you like to see us cover? What FreeBSD related topics puzzling you could we help with? Feel free to get in touch with your ideas.


Contributed by Benedict Reuschling


The post ZFS automatic snapshots with Sanoid on FreeBSD first appeared on FreeBSD Foundation.

Continue reading...
Click to expand...
I am looking for a tool to replace rsnapshots for my UFS system. I plan to perform migration to ZFS

THANKS
 
It's worth taking a look at zrepl. I used Sanoid for a number of years but I have been using zrepl for a few months now. I like how I don't have to think about stopping a replication that is in progress when I reboot a host, it reliably carries on after the reboot. I use zrepl to create replicas of my ZFS filesystems that are sent to a dedicated replication host. These run roughly every hour, but are spaced out over the hour to give each server a timeslot for replication. The SAS drives on the destination host are writing continuously during the day. Overnight, that server replicates the dailies to another replication host that powers up for the task and switches off when it is done.
 
There’s actually quite a few tools out there, and they all work to some extent. Some do snapshots and replication, some do only snapshots.

I haven’t tried Sanoid, but if I ever move my NAS(s) to vanilla bsd, that is what looks most appealing to me.

For now I use a modified version of https://github.com/aaronhurt/zfs-replicate which I have renamed.

If you want to have a look, it is at https://github.com/tschettervictor/xync

Reasons why I modified it are given in the repo.
 
I like the idea of technical posts (keep 'em comming)!

I follow along but seems a bit more complicated then it needs to be (isn't it? ...maybe. just me?).

I mean, the ansible tool seems a bit sledge-hammery to me (install a package and set up a crontab). The 'sanoid' tool also seems a bit heavy-headed ("hammer like")--their video was cute though. I use 'zfsnap' which seems a bit more lighter (BSD like), but the choice of zfs snapshot tools isn't really the point to my post. My point was more along the lines of: Why not showcase how you can elimate all/most of the cruft with FreeBSD? Use ssh, and /etc/cron.d for example (a lot more machines have ssh, then do ansible I would imagine). Show me how I can do this on my "(grand)parent's laptop" without having to install extra software.
 
Haha, interesting point.

With the “xync” script I shared, that is literally all I do.

It is a config file and a binary, which I run via cron everyday day at 2am.

The only thing you need to change in the config file is your source and destination.
 
Mine was an 'interesting point'? ...I'm somewhat serious here: it used to be (at least it seems is used to be from what I read) a point of pride in the BSD/unix community to come up with the simplest, most elegant solution. Using "yet another markup language" (which isn't what YAML actually stands for, BTW) along with yet another tool isn't elegant or simple IMO.

For example I feel as though we need articles that give us more knowledge than what we asked for about the engineered FreeBSD system and I can demonstrate something along these lines with what I stated above.

I mentioned that I use /etc/cron.d (or: /usr/local/etc/cron.d) and this is another example of simple. I create a cron.d file that is structured/named in a way that I can search for easily via scripting.

Code: 
 % find /usr/local/etc/crond.d -type f -name "zfsnap*" -print

My crond.d files look like this:
Code: 
# /usr/local/etc/cron.d/zfsnap_create.crontab - zfsnap_create crontab
# vim: ft=crontab
#
# This will run zfsnap (create) every Friday at 10:00am.
# Snapshots are kept for 14 days.
# 
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#
#minute    hour    mday    month    wday    who    command
0       10      *       *       5       root   /usr/local/sbin/zfsnap snapshot -a 14d -r zroot
And you can infer, or the author can mention, what the SHELL and PATH variables can be used for which can be nice tools in your toolbox.

Then we can talk about how these files could easily be written via a script or copied up with SCP.

The below is obviously not that solution but I placed a few echo commands where you can replace with a 'pkg install' and whatnot into a SSH command that demonstrates how to issue commands on a remote machine as root.

Code: 
ssh -l user -i ~/.ssh/id_rsa server.local -t 'su -m root -c "(echo 'I am root.'; echo 'This is another command.')"'

The -t switch to ssh is really a powerful little command option in this case. Take it out and see what happens.

Obviously, I typed this up here (and didn't spend any time polishing it like a real tutorial would be) but hopefully, you still get the idea.
 
You must log in or register to reply here.
Back
Top