Backup solution for ginormous ZFS pool?

Terry_Kennedy said:
I'd be interested in this when it is ready - as I mentioned elsewhere in the thread, I will have an off-site system with identical hardware for redundancy.
Hi Terry,

You might want to check out this thread. I've finally finished a set of articles that describe what I had in mind. Maybe it is of use to you.

Although 22TB data is several times more data than what I have, you could buy 7+2 Hitachi 3TB HDDs for a RAIDZ2 setup and it would cost $1260. I realize you probably already have tape drive (starting at $1500 on newegg). As a comparison, if you have 5 free SATA ports that support port multiplication, you only have to buy 5* e-SATA docks for something like $15/ on ebay plus some SATA data + power connectors that would cost a total of about $100 all up. For each backup HDD pool, you'd want silicone covers for each HDD ($4 per HDD) and a padded camera bag for $60-$80.

You could probably have 3 backup HDD pools before tape starts being cheaper.
 
carlton_draught said:
You might want to check out this thread. I've finally finished a set of articles that describe what I had in mind. Maybe it is of use to you.
That's quite interesting. I've skimmed it and will take some time to look at it in more detail.

Although 22TB data is several times more data than what I have, you could buy 7+2 Hitachi 3TB HDDs for a RAIDZ2 setup and it would cost $1260. I realize you probably already have tape drive (starting at $1500 on newegg).

You could probably have 3 backup HDD pools before tape starts being cheaper.
I'm probably being old-fashioned, but I want to have some sort of media that I can restore bits and pieces from, onto any sort of filesystem. I don't want to have to depend on ZFS for both my primary data and backups.

I had a problem early on where my ZIL SSD failed (it is a PCI Express card with flash daughterboards, and one of the daughterboard sockets failed). Fortunately, I was able to copy the data off the pool to another server (plus, I had a tape backup which would have been slower to restore). Any attempt to write to the pool crashed the system (post-8.2-RELEASE 8-STABLE).

I'm running 8-STABLE + ZFS v28 on two of the 32TB boxes and 8-STABLE with the "stock" ZFS v15 on the third. It is possible that the pool with the corrupted ZIL would have been fixable if it was at v28.

Code:
(0:867) new-gate:~terry# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
[snip]
rz1:/data              21T    9.5T     11T    44%    /var/www/docs/media/data1
rz1m.offsite:/data     21T    9.6T     12T    44%    /var/www/docs/media/data1m
rz2:/data              21T    9.5T     12T    44%    /var/www/docs/media/data2

I have a Dell PowerVault TL4000 with (currently) a single LTO4 drive. That gives me 44 slots (48 if I don't care about the import/export mailbox) worth of tapes. New-with-warranty LTO4 media is about $26/tape in reasonable quantities. According to Amanda, the average tape capacity (compression enabled) is about 820GB. That's about what I'd expect, given that the ZFS pool only gets about 1.01 compression and the LTO4 native capacity is 800GB, which costs me something like 3.2 cents/GB. Plus, I have nice DataGuard tape cases which hold 20 tapes to move them around to the various offsite storage sites.

22TB is the usable size of my pool, which joins 3 5-drive raidz's with a spare drive for a total of 16 drives, all 2TB:

Code:
zpool create -f data \
raidz label/twd0 label/twd1 label/twd2 label/twd3 label/twd4 \
raidz label/twd5 label/twd6 label/twd7 label/twd8 label/twd9 \
raidz label/twd10 label/twd11 label/twd12 label/twd13 label/twd14 \
spare label/twd15 log label/ssd0
#
zpool set autoreplace=on data
zfs set dedup=on data
zfs set compression=on data

Those are all WD RE4 drives. I realize that I could use cheaper drives for backup. The least-expensive 2TB drive on newegg is currently a Hitachi, at $69.99 [the rebate doesn't help, unless they're going to give rebates on bulk purchases], which comes in at around 3.4 cents per GB, which is more expensive than the LTO4 tapes. Of course, it doesn't factor into account the cost of the tape library - OTOH, it also doesn't factor in the cost of additional SATA controller ports needed for the drives, plus mounting.
 
Terry_Kennedy said:
That's quite interesting. I've skimmed it and will take some time to look at it in more detail.
Thank you.
I'm probably being old-fashioned, but I want to have some sort of media that I can restore bits and pieces from, onto any sort of filesystem. I don't want to have to depend on ZFS for both my primary data and backups.
Fair point. That's why I recommend having 3+ backup pools. If you are reinstalling a whole system, you're starting from that article set + scripts + LiveDVD + fixed/new system, so any issues with the existing system should be a bit of a moot point. And if your current system is usable (e.g. you are just restoring an old file or something, the snapshot scheme will give you a high likelihood of finding it on your system). Of course, you'd want to thoroughly test and understand such a system before trusting your data to it.

I had a problem early on where my ZIL SSD failed (it is a PCI Express card with flash daughterboards, and one of the daughterboard sockets failed). Fortunately, I was able to copy the data off the pool to another server (plus, I had a tape backup which would have been slower to restore). Any attempt to write to the pool crashed the system (post-8.2-RELEASE 8-STABLE).
Yes, this is why if I ever run a ZIL, it will probably be a triple mirror and using something like the Intel 320 series with capacitors for writing buffer data in the event of a power failure, because it's not something that can really afford to fail. L2arc by contrast are great in that you can use a single drive with no problems at all.

I'm running 8-STABLE + ZFS v28 on two of the 32TB boxes and 8-STABLE with the "stock" ZFS v15 on the third. It is possible that the pool with the corrupted ZIL would have been fixable if it was at v28.
Perhaps. I'm going to wait until 9.0-RELEASE before using v28, exciting though dedup may be.

Those are all WD RE4 drives. I realize that I could use cheaper drives for backup. The least-expensive 2TB drive on newegg is currently a Hitachi, at $69.99 [the rebate doesn't help, unless they're going to give rebates on bulk purchases], which comes in at around 3.4 cents per GB, which is more expensive than the LTO4 tapes. Of course, it doesn't factor into account the cost of the tape library - OTOH, it also doesn't factor in the cost of additional SATA controller ports needed for the drives, plus mounting.
I realize that you already have the hardware, so tape is obviously going to be cheaper for you at this point, and perhaps more convenient. I'm not sure whether you'd have need for the sort of system similar to the one in my article set. However, it would be interesting to see if you can identify any more likely points of failure etc (other than the acknowledged N-way voting issue etc.)
 
carlton_draught said:
Yes, this is why if I ever run a ZIL, it will probably be a triple mirror and using something like the Intel 320 series with capacitors for writing buffer data in the event of a power failure, because it's not something that can really afford to fail. L2arc by contrast are great in that you can use a single drive with no problems at all.
Since I don't have a high steady-state write rate (most access is reads of existing data), and writes are things like backups or file copies from other systems which can be re-done if needed, the possibility of a ZIL failure isn't particularly worrisome, particularly as v28 should allow a clean removal of a failed log, without damaging the pool (other than any data not yet committed to the main pool).

Since ZIL writes are intentionally synchronous, I'd be somewhat concerned about increasing the latency with a redundant ZIL, particularly when using MLC flash on a shared controller. The PCI Express ones tend to be a good bit faster. Unfortunately, most of those target the Windows or Linux environments, and many (most?) of them need proprietary drivers not available for FreeBSD. The ones I'm using are LSI-based and use the normal freebsd mpt driver.

Perhaps. I'm going to wait until 9.0-RELEASE before using v28, exciting though dedup may be.
I upgraded my first pool (test system) to v28 when the patches for 8-STABLE first came out. I upgraded the second pool around 6 weeks into the "MFC after: 1 month" v28 commit to HEAD. Unfortunately, the plan for that MFC seems to have been a bit optimistic. I'd like to see it MFC'd at some point - the more people use it, the better it will be.

However, it would be interesting to see if you can identify any more likely points of failure etc (other than the acknowledged N-way voting issue etc.)
I'm using separate OS drives (a gmirror'd pair of 320GB 2.5" drives). Ideally, those would be behind some sort of hardware RAID controller. My previous design from 6+ years ago kept the OS on the same drives as the data. For various reasons, I wasn't happy with that choice. The separate drives proved to be a good idea as I've done at least one set of full drive swaps on each of the systems (one system got 2 sets of full swaps due to a mis-communication with the manufacturer). It was convenient to be able to pull all 16 drives out at once. I expect the restore was also quite a bit faster than a sequence of 5 3-drive swaps/resilvers.
 
Terry_Kennedy said:
I'm using separate OS drives (a gmirror'd pair of 320GB 2.5" drives). Ideally, those would be behind some sort of hardware RAID controller. My previous design from 6+ years ago kept the OS on the same drives as the data. For various reasons, I wasn't happy with that choice. The separate drives proved to be a good idea as I've done at least one set of full drive swaps on each of the systems (one system got 2 sets of full swaps due to a mis-communication with the manufacturer). It was convenient to be able to pull all 16 drives out at once. I expect the restore was also quite a bit faster than a sequence of 5 3-drive swaps/resilvers.
Ah. You might find something in that article of use then, as it puts (most) OS/applications on the SSD mirror, while the rest is on the HDD where space is cheap. In addition to that, there is an interim backup of what is on the SSD mirror, so you can easily restore that if the root mirror dies for whatever reason.

One thing I'm not sure of is how you would get that to play nice with your v28 pool. I guess you could use a newer version of the liveDVD, though I've only ever played with the RELEASE version.
 
Terry_Kennedy said:
I'm probably being old-fashioned, but I want to have some sort of media that I can restore bits and pieces from, onto any sort of filesystem. I don't want to have to depend on ZFS for both my primary data and backups.

IMHO I think you are being sensible. I have written some scripts to manage hourly/weekly/monthly snapshots and replicate them to a remote system via Ssh. I have them running on a few systems and it runs very well. However if the data was really critical then backup to the same technology has obvious risks. You can't rule out a bug affecting both live and backup systems.

cheers Andy.
 
AndyUKG said:
IMHO I think you are being sensible. I have written some scripts to manage hourly/weekly/monthly snapshots and replicate them to a remote system via Ssh. I have them running on a few systems and it runs very well. However if the data was really critical then backup to the same technology has obvious risks. You can't rule out a bug affecting both live and backup systems.

cheers Andy.
Is the risk contingent upon both systems being live, in effect? Or do you still see a risk when you have 5 or 6+ backup pools that spend most of their lives disconnected from anything, sitting in a firesafe or similar (i.e. offline), with a couple of the pools only updated once every few months so as to guard against a bug that is not overt?
 
carlton_draught said:
Is the risk contingent upon both systems being live, in effect? Or do you still see a risk when you have 5 or 6+ backup pools that spend most of their lives disconnected from anything, sitting in a firesafe or similar (i.e. offline), with a couple of the pools only updated once every few months so as to guard against a bug that is not overt?

Hmm, well you are definitely making a lot of effort to minimise risk in the scenario you describe. I was certainly imagining ZFS replication being made to other online systems. But an offline system using a different system vs and offline system using the same, I would say different is clearly better (without getting into details about each system and assuming both systems are considered production ready). Also in the case of backups, where people are doing for example 1 backup a day or more, I think in the instance of disk to disk backup you won't commonly see people taking those disks offline each day.

As I mentioned, I am using ZFS as backup on some systems too, I'm not saying its bad. But in summery my opinion would be that if the data is really critical, then putting all your eggs in one basket is a sub-optimal backup solution. But then when you are designing a backup solution you are always making choices, based on a requirement which will differ from situation to situation, all of which will have pros and cons and a price...

cheers Andy.
 
Thanks for prompting this discussion, Andy.
AndyUKG said:
Hmm, well you are definitely making a lot of effort to minimise risk in the scenario you describe. I was certainly imagining ZFS replication being made to other online systems.
If we are dealing with critical systems where there is no other record being made of the important data that accumulates during that time, it's probably not a bad idea to replicate online in addition to keeping the offsite, offline backups. Doing that is more of an interim measure that will mitigate the loss of the primary server by minimizing the data loss for little ongoing effort (as it's automated).

However, a strictly online "backup" does not guard against an online disaster for example. e.g. if both boxes are rooted and the cracker decides to secure shred both of your copies. See this post. And read the thread, especially monkeyboy's posts. My aim was to resolve a lot of those issues with my attempt at a solution. To quote monkeyboy:
monkeyboy said:
it ain't a "backup" unless it is 1) full, 2) on removable media, 3) offline, 4) offsite, 5) tested for actual restore...
AndyUKG said:
But an offline system using a different system vs and offline system using the same, I would say different is clearly better (without getting into details about each system and assuming both systems are considered production ready).
I can see that in the case where you have two backup systems (e.g. a straight ZFS system as in my example, and some sort of backup to tape that is not ZFS), it has potential to be even more reliable. This is of course provided that you have allocated the extra funds to things like testing that the restore actually works properly, documenting all your procedures, and so on. Often IRL there are compromises made. Maybe you use less tapes/HDD pools than you would with a single solution. Maybe you don't document them well. Maybe one doesn't get tested properly. In the process of attempting to eliminate that risk one can end up introducing more risk.

But then again, I've always been one to put all my eggs in as few baskets as possible, individually wrapped in bubble wrap, 100 feet under concrete under high ground, with castle walls, a moat, pillboxes with overlapping fields of fire... you get the picture.

And if we compare say, ZFS on regular system plus LTFS on the tapes vs ZFS on regular system and backups, then we also conceivably have two things that can go wrong in the former, where only one thing can go wrong in the latter. I realize that we are probably thinking that "LTFS is tape, it can't go wrong", but really it's just another filesystem. If anything, because something like ZFS is used on live systems, any bugs are going to be noticed and corrected that much sooner.

Filesystems would have to be one of the most extensively tested software in existence, because
  1. Every computer uses at least one filesystem
  2. The filesystem is in use all the time, every time the computer does virtually anything.
  3. When there are bugs, particularly data destroying bugs, people get very, very mad. They WILL let someone know about the bug, and if it's much of a problem they will quickly use something else.
  4. Designers know this, and particularly with filesystems that are designed to be used in servers, especially servers that are on the more reliable end of the scale (e.g. ZFS), they are going to be more conservative, do more testing etc.

Once a filesystem has been used in the field reliably for a reasonable period of time on a reasonable install base, the probability of there being some sort of showstopping, data destroying bugs, especially ones that would somehow not show themselves after something like repeated successful zpool status checks, imports, exports and the like, and then simultaneously render all of your backup pools unusable, even those a few months old or so, while functioning perfectly on your primary system up until that point... in my estimation would be remote. To the point where (/me puts on the dogbert hat) a hand-crafted company destruction script that makes it look like you are making backups when in actual fact you are shredding all your backups until the fateful day when your primary system is destroyed - might be more probable than that scenario. Or even if you use tape, a sysadmin with an axe to grind decides to surreptitiously destroy all your tape archives that are theoretically only written to once, along with all other copies of the organization's data.

I guess the thing to realize with risk management is that try as you might, you can never get the risk to zero. Even if you decide to nuke it from orbit, maybe the aliens are already on board.

AndyUKG said:
Also in the case of backups, where people are doing for example 1 backup a day or more, I think in the instance of disk to disk backup you won't commonly see people taking those disks offline each day.
Where I used to work they would take a tape backup each day, and take them offsite each day in a rotation. It would have made no difference to the ease in which that procedure was done to use HDD instead. Even something like 12 disks in a (padded) camera bag, a woman can still carry that by herself. The way I suggest doing it (if you read all the articles I wrote, you'll find it there in the preface articles) is use standard internal HDDs, put them in the cheap $4 silicone HDD cases (which basically provide some small shock protection, stop them sliding around your desk, allow stacking of them on said desk as high as you'd want (they interlock), and provide access to the data and power ports). Excuse the PATA HDD in the image, we'd use SATA of course.
e.g.

2utncip.jpg


You connect the HDDs (still in the cases) to SATA data + power extenders (otherwise you'd have to remove the silicone cases).

2cen880.jpg


You use dual e-SATA HDD docks, that you connect the above extender to.

5aotid.jpg


You connect the docks via e-SATA to your e-SATA back plates, which are in turn connected to regular internal e-SATA ports on your motherboard or SATA card.

wsmblz.jpg


Making a backup is as easy as:
  1. Put HDDs on a flat surface.
  2. Connect to the SATA extenders coming from each HDD dock (two per dock, obviously)
  3. Turn on each HDD dock.
  4. Wait 10-20 seconds for your HDDs to spin up and detect.
  5. Execute the backup script.
  6. Flick off HDD dock switches when script finishes.
  7. Remove HDDs from extenders.
  8. Stack HDDs in padded bag (e.g. camera bag) to be taken offsite.

AndyUKG said:
As I mentioned, I am using ZFS as backup on some systems too, I'm not saying its bad. But in summery my opinion would be that if the data is really critical, then putting all your eggs in one basket is a sub-optimal backup solution. But then when you are designing a backup solution you are always making choices, based on a requirement which will differ from situation to situation, all of which will have pros and cons and a price...
Exactly.
 
Hi,

I didn't say you couldn't offline disks each day, just that I don't think you will find many people using that system currently. Maybe that will change, but I think in big enterprises people still use tape and virtual tape. In not big enterprise things are propbably more mixed.

Online backup vs offline backup its certainly a valid point that someone can destroy online backups all at once if your server is compromised. That's why in a real rolls royce solution for critical systems, in say banks, you will find 2 or 3 online replicas/copies of data with snapshots and 1 or 2 backups to tape each day/hour.

With regards to introducing more risk by having some other tape system that you need to test and document, well I guess thats true in a sense. Two systems to test and document means double the chance of some human error in one of those. I don't think I believe that is a valid arguement for putting all your eggs in one basket, ie weighing the risks against each other,

cheers Andy.

PS If someone has no experience setting up backups (ie home user or small company), then yeah trying to do too many things you will probably end up in trouble, in that situation better to have one system that works really well. For medium to large companies, I think the argument starts to fall down.
 
AndyUKG said:
Hi,

I didn't say you couldn't offline disks each day, just that I don't think you will find many people using that system currently. Maybe that will change, but I think in big enterprises people still use tape and virtual tape. In not big enterprise things are propbably more mixed.
Definitely true. I was as much pointing out that it's very cheap and easy to use HDD, I guess if anyone else is reading. My point is that now, if you use FreeBSD, you can have what is a very reliable system, and a good means to back up. Certainly not as reliable as possible, but then, you pay for that too. At least, it can be a very good building block for small companies (very high data integrity, cheap, fast, convenient), with room for them to grow.
Online backup vs offline backup its certainly a valid point that someone can destroy online backups all at once if your server is compromised. That's why in a real rolls royce solution for critical systems, in say banks, you will find 2 or 3 online replicas/copies of data with snapshots and 1 or 2 backups to tape each day/hour.
That's a good idea. Tape is certainly good if you can afford it.
With regards to introducing more risk by having some other tape system that you need to test and document, well I guess thats true in a sense. Two systems to test and document means double the chance of some human error in one of those. I don't think I believe that is a valid arguement for putting all your eggs in one basket, ie weighing the risks against each other,
I certainly agree that tape is an excellent and probably superior addition to such a system of HDD backups, if the cost benefit calculations work out.
PS If someone has no experience setting up backups (ie home user or small company), then yeah trying to do too many things you will probably end up in trouble, in that situation better to have one system that works really well. For medium to large companies, I think the argument starts to fall down.
Yes. I guess I am thinking like a small company/startup/medium company that likes to do IT stuff inexpensively. For even some medium sized companies I know, this would be a vast improvement. It depends on your definition of medium and how disfunctional the companies you know are. :)

From what I hear, there have been a lot more cowboys in banks too, in recent times. It's a long way from the post-depression environment where you were judged by your mistakes rather than your (current) successes.
 
carlton_draught said:
Yes. I guess I am thinking like a small company/startup/medium company that likes to do IT stuff inexpensively. For even some medium sized companies I know, this would be a vast improvement. It depends on your definition of medium and how disfunctional the companies you know are. :)

Yep, each person has to aim at the best design within his or her budget. One thing that can be said for people of all budgets; budget for backup from the beginning, it makes no sense to have all the bells and whistles on your servers but then be stuck wit a poor backup solution because you already spent all your money.

cheers Andy.
 
I am playing from time to time with backup solutions like these. Stopped using tapes years ago, because HDDs are cheaper/faster and.. more flexible in the long run. Was using DVDs at one time, by the way :)

Anyway, why not use 2.5" drives? Current 3.5" capacity is 3TB, while the current 2.5" capacity 1TB or more. An 2.5" drive is much smaller/lighter than an 3.5" drive. About as much faster for backup purposes.

By reading your comments, I was thinking of a small multi-disk case with integrated port multiplier (or SAS expander) and multiple 2.5" drives. A 16 disk enclosure should not be heavy or big to carry.

Also, you may use disk drives instead of tapes for "tape" backup -- that is, use the drives as linear storage. With, or without filesystem.

With the mirror-split feature in new ZFS, one could use a different technique -- say have 16 mirror pairs of 1TB, connect 16 external drives and attach each of these to the mirrors, in effect creating triple-mirrors. Then wait for all mirrors to resilver. Split mirrors and you have an identical ZFS pool.

You can do this even remotely with HAST, or ggate. Disks can be connected to the system via the network.
 
Terry_Kennedy said:
ZFS snapshots and send/receive don't address the underlying issue - having one or more complete copies of the data in another location for disaster recovery.
[Side note: This topic is now a year old and is still going strong - please keep going.]

I was doing my regular scan through the lists.freebsd.org mailing list archives for ZFS-related posts, and came across 2 different threads that I'd like to excerpt here. The folks already participating in this thread know how important having some sort of backup is (regardless of what strategy is used), but for people just browsing this topic, these may be interesting reading...

freebsd-stable, May 2011:
Olaf Seibert said:
I moved those directories to the side, for the moment, but I haven't been able to delete them yet. The data is a bit bigger than we're able to backup so "just restoring a backup" isn't an easy thing to do. Possibly I could make a new filesystem in the same pool, if that would do the trick]

freebsd-fs, June 2011:
Bartosz Stec said:
I had so much faith in raidz + snapshots, that I have absolutely no backups of this system :( I don't care about time which is needed to find a problem and possible fix, but I am really worried about my data now. What else could I do?
 
ref: http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html

BACULA

Solaris ZFS/NFSv4 ACLs

This is an upgrade of the previous Solaris ACL backup code to the new library format, which will backup both the old POSIX(UFS) ACLs as well as the ZFS ACLs.

The new code can also restore POSIX(UFS) ACLs to a ZFS filesystem (it will translate the POSIX(UFS)) ACL into a ZFS/NFSv4 one) it can also be used to transfer from UFS to ZFS filesystems.
 
Terry_Kennedy said:
ZFS snapshots and send/receive don't address the underlying issue - having one or more complete copies of the data in another location for disaster recovery.
I know this is an old topic, but I read something today that is extremely relevant. I don't know if they were using ZFS or even FreeBSD, but the lesson learned is relevant nonetheless.

Today I happened to be browsing blu-ray.com and the lead article is titled "Database Loss". I'll paste the lead paragraphs of that article below, but click on the above link for the whole thing.

Perhaps reading the article will help convince people that RAID is not backup, no matter what the implementation. Perhaps not. If it does manage to convince you, good.

www.blu-ray.com said:
We are extremely sad to let you know that we've experienced 7 weeks of database loss. 7 weeks ago we moved to a new much improved server, but unfortunately earlier today the hard drives of the database crashed (was using RAID). On the old server we did daily backups, but since we changed server and setup, the old server backup solution didn't work anymore. We have been discussing the new backup system on a daily basis, but hadn't yet implemented it, so the timing couldn't have been worse.

What is missing the last 7 weeks

DVD database additions/updates
All products except Blu-ray movies
Price trackings
Collection updates
HT Gallery updates
Movie ratings
Movie reviews
All forum activity
New user registrations
Cast & crew
Any other database data other than mentioned below
 
My "big" homeserver is a 12x2 TB raidz2 setup - my backup server is a 9x1.5 raidz + 2x3 partly raidz partly nonraid-zfs-big-volume.

Though zfs send/receive is really nice - I do prefer rsync due to more flexibility - so I can change my backup system e.g. to Linux :O :D if there is some exotic hardware that is supported by linux like a backup on stone tablets.

I personally like the flexibility of an rsync backup with easy including/excluding so I can decide which directories go to raidz backup and which into a just plain non raidz zfs (as I have them elsewhere) and it's quite fast - at least faster than gigabit though using low-end hardware :)
 
fgordon said:
Though zfs send/receive is really nice - I do prefer rsync due to more flexibility - so I can change my backup system e.g. to Linux :O :D if there is some exotic hardware that is supported by linux like a backup on stone tablets.

You can use ZFS on Linux as well: http://zfsonlinux.org/ (native) or by FUSE.
 
ZFS on Linux is pretty much a non-starter, the stability of the kernel-land one is lacking and the performance of the user-land one is lacking. I looked at it and decided on FreeBSD, even though I'm a Linux penguin. ZFS on FreeBSD is worlds more stable than ZFS or BTRFS (their currently-buggy clone of ZFS) on Linux.

The correct canonical way to do tape backups is with a tape library, a tape changer driver, and a program such as BRU or Bacula that knows how to invoke the tape changer driver when it hits the end of a tape. That said, recovery from tape backups is always problematic, and even if you do this you likely want to investigate some better option for normal recovery operations (the kind where a file server goes kablooey and you want to revive its contents, not the kind where your building burns down). At which point ZFS's replication capabilities become *very* interesting...
 
I had problems using zfs send/receive to send 1 TB to a remote. I decided to break the single 1TB filesystem to contain several children. Now after a network failure at worst only a child needs to be resent. I use my script to take care of recursive sends and to keep the remote in sync: https://github.com/dareni/shellscripts/blob/master/zfsDup.sh

I finally polished this script today I hope it can be of use to someone else.

Example output:

Code:
# zfsDup.sh shelltests
Test: array_add()
Test: array_clear()
Test: array_iterator()
Test: nameValidation()
Test: isValidSnapshot()
Test: getSnapshotFilesystems()
Test: getSnapshotData()
Test: getRemoteDestination()
Test: printElapsed()
Test: convertToBytes()
Shell tests completed, check the output for errors.

# zfsDup.sh zfstests
Start zfs tests.
Test: new parent file system.
Test: new child file system.
Test: simulate a failed send of the child filesystem.
Test: duplicate and check the child@2 snapshot is resent.
Test: snapshot existing files with updated child data.
Test: simulate a fail send os child@3
Test: snapshot test1.
Test: snapshot test2.
Test: snapshot test3.
Snapshot tests completed ok.
Test: remote host free space.
Test: new remote FS with no quota.
Test: incremental remote FS update with no quota.
Cleaning up zroot/tmp/zfsDupTest/dest zroot/tmp/zfsDupTest/source
Test execution time: 89secs
ZFS tests completed, check the output for errors.


# zfs list -t all -r ztest
NAME  USED  AVAIL  REFER  MOUNTPOINT
ztest  344K  448M  19K  /ztest
ztest@1  9K  -  19K  -
ztest@6  9K  -  19K  -
ztest/backup  112K  448M  19K  /ztest/backup
ztest/backup@1  9K  -  19K  -
ztest/backup@2  0  -  19K  -
ztest/backup@3  0  -  19K  -
ztest/backup@4  9K  -  19K  -
ztest/backup@5  0  -  19K  -
ztest/backup@6  0  -  19K  -
ztest/backup/data  57.5K  448M  20.5K  /ztest/backup/data
ztest/backup/data@1  0  -  19.5K  -
ztest/backup/data@2  0  -  19.5K  -
ztest/backup/data@3  9K  -  19.5K  -
ztest/backup/data@4  9K  -  19.5K  -
ztest/backup/data@5  0  -  20.5K  -
ztest/backup/data@6  0  -  20.5K  -

# zfs list -t all -r zroot/tmp
NAME  USED  AVAIL  REFER  MOUNTPOINT
zroot/tmp  38K  443M  19K  /tmp
zroot/tmp/zfsDupTest  19K  443M  19K  /tmp/zfsDupTest

# zfsDup.sh ztest zroot/tmp root@localhost
================================================================================
Starting duplication 20151001 16:10:56 ...
ztest@6...new...19K...0hr.0min.1sec
ztest/backup@6...new...19K...0hr.0min.1sec
ztest/backup/data@6...new...20.5K...0hr.0min.0sec
Duplication complete 20151001 16:11:04.
================================================================================

# zfsDup.sh ztest zroot/tmp root@localhost
================================================================================
Starting duplication 20151001 16:11:25 ...
ztest@6...up to date
ztest/backup@6...up to date
ztest/backup/data@6...up to date
Duplication complete 20151001 16:11:29.
================================================================================

# zfs snapshot -r ztest@7
# zfsDup.sh ztest zroot/tmp root@localhost
================================================================================
Starting duplication 20151001 16:12:25 ...
ztest@7...incremental...9K...0hr.0min.1sec
ztest/backup@7...incremental...9K...0hr.0min.1sec
ztest/backup/data@7...incremental...10K...0hr.0min.0sec
Duplication complete 20151001 16:12:33.
================================================================================

# zfs list -t all -r zroot/tmp
NAME  USED  AVAIL  REFER  MOUNTPOINT
zroot/tmp  124K  442M  19K  /tmp
zroot/tmp/zfsDupTest  19K  442M  19K  /tmp/zfsDupTest
zroot/tmp/ztest  86K  442M  19K  /tmp/ztest
zroot/tmp/ztest@6  9K  -  19K  -
zroot/tmp/ztest@7  0  -  19K  -
zroot/tmp/ztest/backup  58K  442M  19K  /tmp/ztest/backup
zroot/tmp/ztest/backup@6  9K  -  19K  -
zroot/tmp/ztest/backup@7  0  -  19K  -
zroot/tmp/ztest/backup/data  30K  442M  20K  /tmp/ztest/backup/data
zroot/tmp/ztest/backup/data@6  10K  -  20K  -
zroot/tmp/ztest/backup/data@7  0  -  20K  -
 
Back
Top