Setting up FreeBSD with Auto ZFS snapshots

churchi

Member

Reaction score: 4
Messages: 58

Well after issues with having some important files on my ZFS pool with out any backups or snapshots, i was wondering how i would go about setting up auto ZFS snapshots in FreeBSD?

What i would like to achieve is to enable a snapshot once a day and have 30 copies (1 month) then delete the oldest after one month to rotate the snapshots. Is this possible in FreeBSD? If so would someone be so kind as to share this with me and how i would achieve this.

I will be setting up another machine to mirror my current server for backups, however i am really interested in the snapshots that ZFS can provide for my file system.

If anyone has some setups that i could try to implement to automate this, then i would be really appreciative if you could share

thank you.
 
  • Thanks
Reactions: sdf

jalla

Well-Known Member

Reaction score: 85
Messages: 385

This gives me an opportunity to recommend Ralf Engelschall's exellent snapshot utility sysutils/freebsd-snapshot. It integrates nicely with periodic(8) for scheduling and amd(8) for on-demand mounting of snaps. Works both with zfs and ufs.

It's saved my ass more than once, like when I overwrote my kernel with one that was unbootable:)
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

lme@ said:
ruby.... come on....
Wrote this in few minutes:
http://hg.bsdroot.lv/aldis/zfSnap

it's designed to be used with cron.
By default (age value) is set to keep snapshots for 30 days
you pass zfs filesystem names as args, that you want to make snapshots
if arg1 is -r then zfSnap will make recursive snapshots

snapshots will be save like:
Code:
a/home@2010-07-08_20:04:02
%Y-%m-%d_%T

how do you like it?
I haven't fully tested it yet.... but for now it looks good
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

killasmurf86 said:
how do you like it?
I haven't fully tested it yet.... but for now it looks good
What it's lacking IMO (apart from UFS support of course) is the ability to keep a various number of snaps depending on age. IE, what I want from an automated setup is to take snapshots every couple of hours and keep them for a couple of days, then keep one daily snapshot up to a week and one weekly snapshot for, let's say, one month.

It would definitely take more than a few minutes to write, at least for me, but then fortunately such tools already exist in ports. Even in plain sh(1) if you're so inclined.
(I'll refrain from applying my newly learned knowledge of the port tag here, but I guess you catch my drift:))
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

jalla said:
What it's lacking IMO (apart from UFS support of course) is the ability to keep a various number of snaps depending on age. IE, what I want from an automated setup is to take snapshots every couple of hours and keep them for a couple of days, then keep one daily snapshot up to a week and one weekly snapshot for, let's say, one month.

It would definitely take more than a few minutes to write, at least for me, but then fortunately such tools already exist in ports. Even in plain sh(1) if you're so inclined.
(I'll refrain from applying my newly learned knowledge of the port tag here, but I guess you catch my drift:))
simply add to your cron job either small script that will take snapshots with different pattern, or simply add entry per fs that you want to make snapshot for
for example
/etc/crontab
Code:
# zfs snapshots 2x a day
45      6,18    *       *       *       root    /usr/local/bin/zfSnap.sh -r tank
10      30      *       *       1       root    zfs snapshot -r tank/home@keep_`date +%Y-%m-%d_%T`
This will make snapshots of all file systems 2x a day
and 1 snapshot of /home and all it's sub-file-systems every Monday
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

Firstly thank you all very much for the replies. I am hopeful from all of these replies that i can put in place a snapshot system that will save me from what happened yesterday.

jalla said:
This gives me an opportunity to recommend Ralf Engelschall's exellent snapshot utility sysutils/freebsd-snapshot. It integrates nicely with periodic(8) for scheduling and amd(8) for on-demand mounting of snaps. Works both with zfs and ufs.

It's saved my ass more than once, like when I overwrote my kernel with one that was unbootable:)
I am liking this port. A few questions on this:
1. Can this be configured to take a snapshot once a day, and just keep for 30 days only?
2. if the above can be done, can the utility auto delete once the 30 days has been up for the oldest snapshot?
3. can the snapshots be recursive if using the above way (deleting the oldest from 30 days ago) or would they have to be full snapshots every day if doing it this way?


>>lme@ sysutils/zfs-snapshot-mgmt

I have also looked into this script however i cant see a way for the script to delete the oldest snapshot after the period i select? if there is a way can someone show me please?
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

try my script ;) it will auto delete snapshots older than, 30 days [those, that this script created or match pattern mentioned above]
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

sweet, let me set it up and give it ago.

Do i need to schedule the weekly full snapshot, or this scrip will keep all needed snapshots for 30 days to recover files, then auto delete them after 30 days?
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

churchi said:
sweet, let me set it up and give it ago.

Do i need to schedule the weekly full snapshot, or this scrip will keep all needed snapshots for 30 days to recover files, then auto delete them after 30 days?
I'm not sure I understand you...
This script will create snapshots, if it's called with arguments (it will make recursive snapshots if first argument is -r).

after snapshots are made, it will check zfs snapshots, that match pattern.
From each snapshot that match pattern it will calculate creating time in seconds from Eposh
It will then compare this time to current time (in seconds from Eposh) minus age seconds.
If snapshot is older, than age seconds, then snapshot will be deleted

If you want some snapshots to remain you need to set crom, or somehow otherwise create zfs snapshots, that won't match patter used to identify zfs snapshots created with zfSnap.sh

Does this answer your question?
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

killasmurf86 said:
simply add to your cron job either small script that will take snapshots with different pattern, or simply add entry per fs that you want to make snapshot for
for example
/etc/crontab
Code:
# zfs snapshots 2x a day
45      6,18    *       *       *       root    /usr/local/bin/zfSnap.sh -r tank
10      30      *       *       1       root    zfs snapshot -r tank/home@keep_`date +%Y-%m-%d_%T`
This will make snapshots of all file systems 2x a day
and 1 snapshot of /home and all it's sub-file-systems every Monday
I'm not convinced

1) Time to keep snapshots is still hardcoded in your script
2) The weekly snapshots aren't rotated automatically
3) You've not deviced any practical method of differentiating hourly/daily/weekly snapshots, except by adding crontab entries to handle every special case

Instead of a shitload of crontab entries, I find it infinitely more flexible (and elegant) with one simple entry in /etc/periodic.conf

Code:
snapshot_schedule="/:4:7:3@11,14"
That's it
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

churchi said:
Firstly thank you all very much for the replies. I am hopeful from all of these replies that i can put in place a snapshot system that will save me from what happened yesterday.



I am liking this port. A few questions on this:
1. Can this be configured to take a snapshot once a day, and just keep for 30 days only?
2. if the above can be done, can the utility auto delete once the 30 days has been up for the oldest snapshot?
3. can the snapshots be recursive if using the above way (deleting the oldest from 30 days ago) or would they have to be full snapshots every day if doing it this way?
With freebsd-snapshot you just put your shedule in /etc/periodic.conf, something like
Code:
snapshot_schedule="/fs:0:30:0"
which means "take a snapshot every day (at midnight), and keep the last 30". Older snaps just "fall off the end".

If you ever find you want to checkpoint your system a few times during the day, just change the schedule
Code:
snapshot_schedule="/fs:0:30:8@10:12:14:16"
Not sure what you mean with 3. but any snapshot is by design "full" in the sense that it represents the complete filesystem at the time it's taken. The actual size of a snapshot though is the difference between that and the live fs.

ps. As many people seem to think snapshots were invented with zfs, I keep stressing the point that they're available with UFS as well (and equally useful).
I should note however that there's a hard limit of 20 snaps per filesystem in UFS.
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

jalla said:
I'm not convinced

1) Time to keep snapshots is still hardcoded in your script
2) The weekly snapshots aren't rotated automatically
3) You've not deviced any practical method of differentiating hourly/daily/weekly snapshots, except by adding crontab entries to handle every special case

Instead of a shitload of crontab entries, I find it infinitely more flexible (and elegant) with one simple entry in /etc/periodic.conf

Code:
snapshot_schedule="/:4:7:3@11,14"
That's it
You can adjust the script to your needs, can't you? and it's really simple to extend it...

and instead of shitload of contab entries, write simple sh script, where you set, how and which zfs file systems you want to backup and exec it as many times as you want...

K.I.S.S.

P.S.
maybe I'll improve things that you noted
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

Hardcoding this stuff in the script is not the way to go. You should factor out all the variables and keep the script itself generic IMO. As to simplicity, a single line in a config file to control all periodic snapshots, which filesystems, what time to snapshot, how many to keep, that's what I call simple.

When you deside to reinvent the wheel you should opt to improve it. No offense, but your "wheel" in this case looks distinctly square to me.
 

graudeejs

Son of Beastie

Reaction score: 696
Messages: 4,616

jalla said:
Hardcoding this stuff in the script is not the way to go. You should factor out all the variables and keep the script itself generic IMO. As to simplicity, a single line in a config file to control all periodic snapshots, which filesystems, what time to snapshot, how many to keep, that's what I call simple.

When you deside to reinvent the wheel you should opt to improve it. No offense, but your "wheel" in this case looks distinctly square to me.
The wheel that I made was especially for particular road.... enough talks....
Check updates, when I improve it ;)
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

jalla said:
With freebsd-snapshot you just put your shedule in /etc/periodic.conf, something like
Code:
snapshot_schedule="/fs:0:30:0"
which means "take a snapshot every day (at midnight), and keep the last 30". Older snaps just "fall off the end".

If you ever find you want to checkpoint your system a few times during the day, just change the schedule
Code:
snapshot_schedule="/fs:0:30:8@10:12:14:16"
Not sure what you mean with 3. but any snapshot is by design "full" in the sense that it represents the complete filesystem at the time it's taken. The actual size of a snapshot though is the difference between that and the live fs.

ps. As many people seem to think snapshots were invented with zfs, I keep stressing the point that they're available with UFS as well (and equally useful).
I should note however that there's a hard limit of 20 snaps per filesystem in UFS.

Cheers thanks guys for all the help. I have decided to give freebsd-snapshot a quick go.

So i have edited my /etc/periodic.conf
and i have the following in there:
Code:
monthly_statistics_enable="YES"
monthly_statistics_report_devices="YES"
monthly_statistics_report_ports="YES"

snapshot_enable="YES"
snapshot_schedule="/storage1:0:30:0"
Do i have to edit anything else in the /etc/crontab for example? or will everything be taken care of by adding in these lines into periodic.conf
Code:
snapshot_enable="YES"
snapshot_schedule="/storage1:0:30:0"
cheers
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

Hi All,

i have setup the above and still can't seem to find any snapshots being taken daily.


Code:
[root@server-01 /home/churchi]# zfs list -r -t snapshot
no datasets available
Is there a better way like adding the above (freebsd-snapshot) to cron? at least i know that cron works on my system.

Thanks.
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

You need an addition to /etc/crontab as well

Code:
0 * * * *       root    /usr/local/sbin/periodic-snapshot hourly
0 0 * * *       root    /usr/local/sbin/periodic-snapshot daily
0 0 * * 0       root    /usr/local/sbin/periodic-snapshot weekly
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

cheers thanks jalla,

so with just adding:
Code:
snapshot_enable="YES"
snapshot_schedule="/storage1:0:30:0"
to /etc/periodic.conf

and adding the above lines:
Code:
0 * * * *       root    /usr/local/sbin/periodic-snapshot hourly
0 0 * * *       root    /usr/local/sbin/periodic-snapshot daily
0 0 * * 0       root    /usr/local/sbin/periodic-snapshot weekly
to /etc/crontab

Setting up this will allow for backup of my ZFS pool of /storage1/

Is that all i need to complete and setup for this to be taking snapshots?

Thanks.
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

In principle, yes. But you must specify the datasets you want to snapshot, not the pool

Code:
snapshot_schedule="/storage1/data1:0:30:0 /storage1/data2:0:30:0"
 
OP
churchi

churchi

Member

Reaction score: 4
Messages: 58

awesome mate. thanks for clearing that up.

so this is my zfs list
Code:
[root@server-01 /etc]# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
storage1              2.23T  8.46T  45.4K  /storage1
storage1/data         2.23T  8.46T  1.46T  /storage1/data
storage1/data/photos   787G  8.46T   787G  /storage1/data/photos
[root@server-01 /etc]# ls /storage1/data/
If i snapshot /storage1/data
will that snapshot /storage1/data/photos
as well?


cheers
 

jalla

Well-Known Member

Reaction score: 85
Messages: 385

No, snapshots taken this way are not recursive. To change that you can modify /usr/local/sbin/snapshot
Code:
gnome:/usr/local/sbin# diff snapshot.orig snapshot
292c292
<                 system zfs destroy "$fs_name@$fs_tag.$i"
---
>                 system zfs destroy -r "$fs_name@$fs_tag.$i"
301c301
<                 system zfs destroy "$fs_name@$fs_tag.$i"
---
>                 system zfs destroy -r "$fs_name@$fs_tag.$i"
307c307
<                     system zfs rename "$fs_name@$fs_tag.$i" "$fs_name@$fs_tag.$j"
---
>                     system zfs rename -r "$fs_name@$fs_tag.$i" "$fs_name@$fs_tag.$j"
313c313
<             system zfs snapshot "$fs_name@$fs_tag.$fs_gen"
---
>             system zfs snapshot -r "$fs_name@$fs_tag.$fs_gen"
This is fine for my usage as I don't want to to do anything but recursive snapshots. Unfortunaly there's no simple way to have the script support both recursive an non-recursive snapshots :(
 
Top