ZFS ZFS to replace rsync as ongoing FS copy mechanism

[George Michaelson]

I have two large ZFS extents on different hosts. They have active filesystems which I have kept in reasonable sync via rsync/ssh. This is hugely inefficient.

I beleive if I was "starting from scratch" I could do this via zfs send/receive pipes (probably using mbuffer(1)) far more effectively: I tried it on another host and the speed difference was impressive.

But I would prefer not to "wipe" the state and start again: I'd like to do some approach which kept my investment in copied data, and established a synchronized state from now onward.

Is this actually possible? or, am I doomed to wiping one, and restarting the copy process to carry forward via zfs send/receive?

BTW, there is a confusion in my mind about how you do this: It looks like incremental ZFS snapshots would be best, but some statements in the FreeBSD ZFS guide say "you need an empty FS to unpack into" which obviously cannot apply to incremental update. And, you want to delete snapshots periodically which on receiver would mean you didn't have a basis to apply ZFS receive increments to, if you don't stay in some lockstep. So I suspect I need to stick to some regime of sending state x, and keeping snapshot state x-1 and only deleting state x-2 x-3 ... ever.

 

[Crest]

Yes you have to transfer and import a full replication stream of the datasets once to use incremental streams later. From ZFSs point of few your datasets aren't related at all. They just happen to contain the same data and similar metadata (if you're lucky). ZFS stores file systems as directed acyclic graphs and a snapshot is a pointer to a root of a graph of immutable nodes. Each node carries a transaction number and an incremental snapshot between two snapshots is basically a stream of all nodes with a transaction number between the transaction numbers of said snapshots.

If you have physical access to both servers, your source and destination pool contain only mirrored VDEVs and you have enough SAS (or SATA) ports you can accelerate the initial copy by attaching replica disks to all mirrors in the source pool and splitting them of into a new pool. That way you can leverage your disk controller bandwidth and avoid clogging your network if all of those conditions are satisfied and you want to replicate enough of the pool to justify hassle of moving physical disks.

 

[da1]

So I suspect I need to stick to some regime of sending state x, and keeping snapshot state x-1 and only deleting state x-2 x-3 ... ever.

That is correct. You need to have the same any 1x snapshot on both machines in order to send the incremental data between that and the latest snapshot.

Ex: SRV1 has snap1. You send it to SRV2. Down the road, you create snap2 to snap10 on SRV1. You now have snap1 on both machines and you can send the diff between snap1 and snap10, from SRV1 to SRV2. You can perform any cleanup you like, as long as you keep any 1x common snapshot (ex: snap7 or snap9) on both of them. You would then send the diff between snap9 and snap31 and cleanup again.

PS: There are many tools to automate this. I personally like to use self-made script, simply because I like writing them. Here is one example: https://forums.freebsd.org/threads/...-continous-zfs-filesystems-replication.28971/

 

[George Michaelson]

I bit the bullet, deleted the zfs and recreated it on the recv side, and I'm close to finished a 17Tb send. I've been taking daily snapshots on the send side, and I'm going to move the increments over when this finishes.

I'm still working out which of the scripted tools to use for future synchronization. A nice thing about snapshotting embedded first class in the filesystem model is that I can safely carry on with a bunch of things in userspace, because the snapshot gives me invariant state on the sender side to work with even while the rest of the world carries on.

I discovered early on that the /pool/filesystem/.zfs/... substructure doesn't like how rsync calls stat() or some other call, and you can't easily use that mechanism to "walk" the hidden state. Its fine for shell level primitive work, but something rsync tries to do in the back didn't work. Doesn't matter: use zfs tools first-class and stop depending on a broken model in userspace is best.

Thanks for guidance! much appreciated.