ZFS Is there a way to REJOIN a split mirror?

R

rowan194

zpool split creates a second, separate mirror.

Is there a way to rejoin the split part to form the original mirror?

To explain further what I want to do:

1. Split a busy mirror
2. Back up (at full speed), using the split device as source, so it doesn't impact the application using the main mirror
3. Rejoin the split device to the main mirror, without needing to do a full resilver (which would be just as bad as doing the backup from the mirror itself)

This could be done manually by offlining one device of the mirror, mounting that device on a separate machine, then returning it to the main machine and onlining it (which will perform a partial resilver)... but I want to automate the process.
Any ideas?
 
Martin Paredes said:
What you are really trying to do is a backup without disturbing the application.

You should use snapshots, read the section Managing Snapshots.
Click to expand...

Replication and incremental snapshots could work, but I'd prefer to stick (if possible) with a file level backup. Having said that, it is pretty painful when rsync has to sift through a couple of terabytes of data on an array where the HDDs are already often at 100% load. Both MySQL and rsync end up competing for disk reads, so when the latter is running, MySQL performance degrades significantly.

My idea was to split the array only temporarily, essentially giving rsync a dedicated drive until the backup was complete, then readding the drive back into the mirror, resilvering only the changes since the split. I can think of a couple of horrible ways to do it (such as: zpool offline -t, ggated the device to another server, import pool as read only on the other server).

Ultimately what I'm looking for is a way to read data from an inactive member. I doubt ZFS would let me read-only mount an offline member with the same pool ID as one already mounted?
 
I don’t think you’ll be able to split one disk off a mirror to read data off for backup, then join it back in.

Personally I’d second the idea of using ZFS snapshots and replication. With anything over 1TB, the idea of checking every file when you can just send an incremental snapshot is a massive difference.

As ZFS knows exactly which blocks are new since the last backup, this will be by far the least abusive on the disks, and unless something like >100GB change per day, will take minutes rather than hours.

of course I would send the snapshot to a second ZFS machine so that you have direct access to all the files, rather than storing the snapshot in one big backup file.
 
How about a mixed solution? Make a snapshot of the original data (quick and painless), transfer the snapshot to another machine and do the file based backup there?
 
usdmatt said:
I don’t think you’ll be able to split one disk off a mirror to read data off for backup, then join it back in.
Click to expand...

FWIW: this is possible, manually, as mentioned in the OP. On another machine, I have a 3-way mirror, with two of the drives being SSD, and the third being a USB HDD that is periodically plugged in to resilver changes. This arrangement ended up saving me a couple of times, when a dodgy power connector on the 2.5" drive bay enclosure caused checksum errors on both SSDs, with permanent corruption of some resting data - plugging in and onlining the third drive was sufficient to self-heal the corrupt data on the SSDs.

Started as a temporary thing when I first had problems with one or both SSDs dropping out of the array, but I ended up keeping it permanently. That's what got me thinking about temporarily splitting an array (of HDDs) to back it up.

usdmatt said:
Personally I’d second the idea of using ZFS snapshots and replication. With anything over 1TB, the idea of checking every file when you can just send an incremental snapshot is a massive difference.

As ZFS knows exactly which blocks are new since the last backup, this will be by far the least abusive on the disks, and unless something like >100GB change per day, will take minutes rather than hours.
Click to expand...

I was looking at various ways to try to slow down rsync's disk access, but I agree it needs to be done at a lower level, since only ZFS has the knowledge required for a true incremental backup. As an extreme example, one of my MySQL tables is over 1TB in size, so updating only a single row will require rsync to scan the entire 1TB+ file (and the entire 1TB+ remote copy) for changes. Even worse, sequential read requests for that 1TB+ table translate to greatly fragmented non-sequential disk seeking, and a seriously low read speed, as per this explanation: https://forums.freebsd.org/threads/...using-nearly-50-more-space.71209/#post-442593

What I'd really like is some hybrid of ZFS and rsync, which uses information about ZFS updated blocks to determine changed data to send, but modifies a remote file rather than a file system. Essentially, replicating the history of write() requests for each file, to the copy at the remote end. I can dream. :)

(I'd also prefer to send to a directory tree on a remote server, rather than needing to maintain a separate ZFS dataset on that server.)

SirDice said:
How about a mixed solution? Make a snapshot of the original data (quick and painless), transfer the snapshot to another machine and do the file based backup there?
Click to expand...

Not really understanding this suggestion, as transferring an entire snapshot would still require all data to be read from the array?
 
rowan194 said:
Not really understanding this suggestion, as transferring an entire snapshot would still require all data to be read from the array?
Click to expand...
Transferring a single snapshot is a lot less I/O intense than the scan rsync does. So yes, there would still be some I/O, but it should put less of a strain on the system.
 
Just a thought on the practicalities of this; you want to split the mirror such that a replication activity doesn't negatively effect the application your running, but won't removing part of the mirror impact read requests? And while a partial resilver is better than a full one, you'll get another hit to your application when you rejoin the mirror...

Having said that, I think that Allan Jude may have mentioned this on the BSDNow Podcast at some point. Might be worth heading to IRC or writing into the podcast to see if you can ask him?
 
idk i did exactly what sirdice said , but i'm kind of lazy.. so i created a rolling snapshot script.. cron tab it and once per day all my servers are magically backed up faster than rsync can generate a file list..

idk in theory you could snap shot it even faster and just use zfs send to keep it uptodate in near real time..

never tried it that quickly but in theory it should work fine..

then you can delete rsync forever.
 
You must log in or register to reply here.
Back
Top