Solved Backup strategy

fred974

fred974

Hi,
Is it foolish to use zfs snapshot as a backup plan?
I have a firend who told me that zfs snapshot is a very poor way of doing backup and I will be better off using something like rsync.

I have 1 physical server that run FreeBSD 10.2 with ZFS filesystem and within it I have around 12 iocage jails.

On the FreeBSD host I run zfs snapshot zroot@bckdate and for the jail I run
iocage snapshot <bignumber>@bckdate.

Could someone please sugest a good way of implementing backup on FreeBSD?
Should I be using commercial solution or do my own?

My biggest worry at the moment is making sure the email server is backup every day
 
getopt said:
1. If the ZFS pool gets corrupted.
2. If the harddisk fails.
[...]
You cannot consider a backup a backup if it is located on the same storage medium.
Click to expand...
Well, if you really only consider these two points, snapshots for preventing human error or corruption as the result of a crash could be good enough, given that:
1. you use (and: trust) scrubbing
2. you use some level of redundancy, e.g. RAID-Z1 if you decide two failed drives at (roughly) the same time is unlikely enough not to care about it.
 
ZFS snapshots are a valid way to backup your data but storage of the snapshots on a separate backup media is more complicated than you would first think. The zfs send process assumes that receiver immediately checks the stream for corruption and that condition is fulfilled by zfs receive if the destination is another ZFS pool. In case any error zfs send | zfs receive pipeline will abort with a clear error and the destination dataset gets rolled back to previous state. If you do not use zfs receive but store the streams as files you might have a problem if there's any corruption in the streams, even a single bit flipped might render the whole stream unusable. If you choose not to use another ZFS pool as the back up media you might be better off with net/rsync because the complexity is much lower and you don't have worry about the data integrity issues as much as you would with the stored ZFS streams.

I don't want to sound like broken record but RAID is never a backup, never.
 
kpa said:
ZFS snapshots are a valid way to backup your data
[...]
I don't want to sound like broken record but RAID is never a backup, never.
Click to expand...
It's not and it wasn't what I said. But a local snapshot wouldn't be very useful as a backup at all, if there wasn't some redundancy to protect the storage itself.

All in all, I would consider the combination of the RAID-Z2 (protecting from hardware errors) and local snapshots (protecting from human error and/or crashing systems) already quite decent ... depending on your requirements of course. If you bring catastrophic events to the table, there's no alternative to getting your backups off the machine (better yet, out of the building), of course.
 
Ok ...
So if I do zfs snapshot and send them to a zfs vps using zfs receive, will it be enough to recover my data if my server goes in flame?
Please understand that this is the very first time I am looking at backup solution for an entire server..
I have script that create mysqldunp and save the webfile but I don't know how to save individual mailboxes for example.
 
Either there's a miscommunication here, or some very important points are being overlooked:
  1. No copy of any data---absolutely none---is a "backup" if the copy and the original are in the exact same place. Your fancy RAIDZ array won't mean anything if your machine catches fire or gets stolen or lightning strikes the electric lines to your house or something. Or more likely, make some maintenance mistake or end up with an intrusion that you couldn't foresee. A snapshot is not a "backup." It's just a safety precaution to keep you from accidentally deleting something.
  2. Creating a proper backup means copying the data to another medium, and keeping multiple backups from different points in time in case one backup fails. With zfs send | zfs receive, this can only be done if snapshots are used. Otherwise every time you make a backup you'll be copying every single file on the origin, and overwriting every single file on the destination.
So no, ZFS snapshots are not backups, but in order to create good ZFS backups you need to use snapshots. They are necessary, but not sufficient. My personal recommendation is to use sysutils/zxfer to back up to a spare disk or server. The first example in the zxfer man page works very similarly to rsync---it will replicate the entire pool, including snapshots, the first time you run it. After that, each time you run that same command zxfer will delete any stale snapshots from the destination that don't exist on the origin, then copy all the new files and snapshots to the destination. You'll always have an identical copy of your pool as a backup, the amount of data copied is minimal, and snapshots will contain any past versions of files. If you back up to a spare disk, and recreate your server's partition scheme on that disk, you can always have a spare boot-able disk in the event of total catastrophe.

Remember: You never need a backup until you don't have one.
 
I think that this is the time to hijack this thread because fred974 is referring to our product which was sold and installed to a jail on his server.
  • It was recommended to Fred that he should implement an offsite backup policy.
  • The mailbox paths were clearly indicated to Fred.
  • We also suggested that database backups should be performed using mysql tools.
Finally, a quote was given to Fred in order for us to implement the backup policy given the fact that he would provide a different machine for us to store his data.
Our company can not guarantee the integrity of data for in premise installations since the management is beyond our control.

Regarding our cloud services, all of our data is stored in Zurich and we replicate them to a DR location in Geneva. Although our service provider is using redundant DC's we chose to replicate everything to a different DC location.

George
 
gkontos ,
I know exactly what you offered and this not an issue as I am planning on going ahead with the solution you offer. But this only provide backup for the email and I have another 8 jails to backup. So this is why I am posting here.
So please don't be offended for me trying to understand how to deal with the rest of my system.
Unless you quote me for a solution that protect the whole server and the next jail I will add.
Commercially I have only used you service and not planning to go anywhere else.
So please call me so we can talk
 
ANOKNUSA said:
No copy of any data---absolutely none---is a "backup" if the copy and the original are in the exact same place.
Click to expand...
That's discussing the meaning of words and there are good reasons to take a "backup copy" physically away, you probably named all of them. Nevertheless, combining redundancy with regular local snapshots already protects you from a lot of things, and, depending on your actual needs, you might judge this to be enough (say, on a private home server where users might just mirror their files, an extended downtime won't cost massive amounts of money and a catastrophic failure will only cost your precious time to reinstall the system).

But, of course, having learned from the latest two posts that this is indeed some kind of commercial production server, yes, there's no alternative to backups in a safe place.
 
The best way to backup a ZFS filesystem is to use "zfs send" to transfer snapshots to another system running ZFS. :) Preferably one that's in another building. It's one of the nicer features of ZFS. Defence in-depth is very useful here.

Create snapshots so that you can recover files locally if the files get deleted or corrupted.

Copy the snapshots on another server in case your primary pool gets corrupted, the server hardware dies, the building burns down, etc.

And (for the truly paranoid), copy the data off to removable storage (CDs, DVDs, Blu-Ray, tapes, external harddrives, whatever) that is stored in a 3rd location, preferably in a different town. :)


If you have access to two systems running ZFS, then a simple script like the following is all you need (for a single filesystem; you can wrap it in a for-loop with a list of filesystems if needed), run through cron every day:

Code: 
#!/bin/sh
yesterday=$( date -v -1d "+%Y-%m-%d" )
lastsnap=""

# Get the last snapshot date
lastsnap=$( ssh -i /path/to/key.file username@remoteserver "{zfs list -o name -t snapshot -r remotepool/remotefs | tail -1" | awk -F'@' '{ print $2 }' )

# Send all snapshots between ${lastsnap} and ${yesterday}
bbcp -4 c -N io "zfs send -I localpool/localfs@${lastsnap} localpool/localfs@${yesterday}" username@remoteserver:"zfs recv -d remotepool"

That uses sysutils/bbcp (needed on both systems) to send a multi-threaded stream of bits across a known-secure network, and will easily saturate a gigabit link. If you have a slower link, or you're transferring over the Internet, then you should use normal SSH to encrypt the traffic. Change the last line to just:

Code: 
zfs send -I localpool/localfs@${lastsnap} localpool/localfs@${yesterday} | ssh -i /path/to/key.file username@remoteserver "zfs recv -d remotepool"

If your off-site backups system isn't using ZFS, then you can just wrap rsync into a script (shameless plug) to copy the data from the ZFS filesystems to the remote server.

No need for fancy software, no need to spend money on software. :)
 
fred974 said:
Is it foolish to use zfs snapshot as a backup plan?
I have a firend who told me that zfs snapshot is a very poor way of doing backup and I will be better off using something like rsync.
Click to expand...
How paranoid you need to be about backups depends a lot on what is being backed up, and what the impact of losing that data forever is.

* If the data is random "media files" downloaded off the Internet, then you can probably replace them (or get different ones) without too much trouble, then the cost of even a single backup might be considered excessive. If the original and all backups are lost, you just have to go download some more "media files" from the Internet.
* If the data is of personal value to you, then an occasional copy to an encrypted external drive you keep at your cousin's house might be sufficient. If the original and all backups are lost, you may no longer have pictures of your wedding, kids, etc.
* If the data is something like the source code to your best-selling utility, you want multiple copies at multiple locations, presumably encrypted. If you lose the original and all backups, your company is either out of business or your brand image is seriously tarnished.
* If the data is personal info (health records, material usable for identity theft, etc.) then your company is probably out of business and you have government investigators investigating, threatening fines, and so on.

Once you've established how important it is to have one or more backups, you now need to assign priorities to various scenarios and see what sort of recovery can be attempted with the backups you have.

* You have a single ZFS pool and rely on snapshots for backups. The power supply in your system fails and decides to zap all of your disk drives. Whoops, no backups.
* Same as above, but a software bug or power failure corrupts the pool. Maybe you can recover some data, maybe not.
* Taking the above scenarios to heart, you splurge on a second system with an equal-or-larger ZFS pool. Maybe you use a different operating system, different brand of hard drives, etc. to try to make the systems as different as possible to eliminate the possibility of a fault on one system taking out the second system. But a tree falls on your house, or your server room gets flooded during a storm.
* You decide to get a very expensive data circuit (or string your own fiber through the woods) to a second site, and set up the second system there. But a hurricane comes through and damages both sites. Or some bug, malicious activity, or simple mistyped command damages your "master" pool and your replication faithfully copies the damage to your backup site.
* Same as above, but substitute "cloud backup" for "second site".
* You decide the best way to be safe is to have the data in a universal format that can be understood by most *BSD/Linux systems, regardless of whether they support ZFS. You make tape backups of the entire pool (not incrementals) and store them offsite in a secure facility.

The above is designed to provoke thought and thoughtful discussions on what an appropriate backup (and recovery - never forget recovery!) strategy is.

In my case, on my home RAIDzilla 2.5 storage appliances, I have multiple tiers of backup:

o Daily local replication to a second, identical system in the same cabinet (useful for recovering from "oops" errors).
o Daily replication via 10GbE private fiber to a second, identical system located a few miles away, on a different electrical substation and on the other side of the hill I'm on. That site can serve as a disaster recovery site if needed.
o Weekly backups of the entire pool (in GNU tar format) to LTO-6 tape media via a robotic library. The media is then removed and placed in special hermetically-sealed tape transport cases, a seal applied, and it waits for the courier to pick it up and store it at a secure facility several states away. Some backup sets stay there "forever", while others come back and are re-used.
 
Hi,
Now, I want to weigh in...............
First some history.........
I am a retired systems engineer who started on IBM1130s proceeded to CDC Ciber 6600, PDP-11s running RT-11, RSX-11M, RSTS-E, VMS, IBM mainframes, OS-2, MPEV, System3, System7, SYSV, DOS, various RealTime OSes, XP, etc.
For the past 20 years I have run various Linux flavors on our shipboard computers. These are subject to random crashes, mostly consequent to emergency starts of the main engine to avoid navigational hazards. I have explored JFS, Ext3-4, ReiserFS, VFAT, NTSF-3g, and others. Only JFS has provided 100% seamless recovery of the crashed system, with at most loss of un-committed records.
Regardless, I have partitioned my disks, imaged the data, separated data from system files, etc.
I don't know ZFS..... and what I am hearing here makes me very UNCOMFORTABLE.
I am running a desktop system not a server farm.
What I want is clear, unequivocal, proven ability to write all files from a system using a live system such as systemrescueCD, to another partition, disk drive, or removable media, including the ability to verify the archive.
So.............
Specific instructions please............ Keeping in mind that I'm now a 69 year old, stroke survivor...... ie; not a kid anymore

Also, since I spend most of my time studying fish and corals, would help if the moderators email me a heads up when a reply commeth.............[Mod: email redacted]


INDY
 
Also, since I spend most of my time studying fish and corals, would help if the moderators email me a heads up when a reply commeth.............[Mod: email redacted]
Click to expand...
At the top of the thread (right-hand side) is a button "Watch Thread". That will inform you if somebody replies to the thread. Posting an email address on a forum is a guaranteed way to get a lot of spam.
 
You must log in or register to reply here.
Back
Top