ZFS zfs send/receive of zroot hanging

stratacast1

stratacast1

I'm trying to do a simple zfs send/receive of a snapshot of zroot, but it seems to halt entirely when sending /usr/ports. What I did:
Code: 
zpool create zrootbackup -o altroot="/mnt/zrootbackup" /dev/da0
zfs create zrootbackup/servername
zfs set readonly=on zrootbackup/servername

zfs snapshot -r zroot@`formatting for date`

zfs send -R zroot@today | zfs receive -dvuF zrootbackup/servername
This is being sent to a 16GB flashdrive. Could it be predicting that I may not have enough space and silently failing and hanging?
 
Why backup /usr/ports/ at all? It's easy to restore without using backups. It's just going to use up space, especially /usr/ports/distfiles ( and any left-over */work/ directories).
 
That's been my thought, but how could I leave out a zfs dataset in a recursive snapshot to be sent? I'm having troubles finding information on that
 
I guess that will work. I was hoping for an easier route without having to iterate each one, but I guess that will be my only choice and I'll have to write a script for that
 
ZFS properties are also great for specifying what should be backed up and even when/how often.

Just add a custom property like e.g. "replicate" to all datasets you want to send|receive to your backup server. Then just iterate over all datasets (CAUTION: this can be _very_ slow for systems with many datasets! Generating a list and writing it to a file might be wise in these cases) and send|receive only those with "replicate=yes" set.

Also have a look at sysutils/zfs-replicate for replication and sysutils/zfsnap2 or sysutils/zfstools for maintaining snapshots. The latter one also uses zfs properties for snapshot scheduling.
All are basically just a collection of shellscripts, so if they don't work for you even by combining them, they can also be used as a source of ideas/code to built your custom solution.
 
sko said:
ZFS properties are also great for specifying what should be backed up and even when/how often.
Also have a look at sysutils/zfs-replicate for replication and sysutils/zfsnap2 or sysutils/zfstools for maintaining snapshots. The latter one also uses zfs properties for snapshot scheduling.
All are basically just a collection of shellscripts, so if they don't work for you even by combining them, they can also be used as a source of ideas/code to built your custom solution.
Click to expand...

sysutils/zap can manage both rolling snapshots and replication, optionally with ZFS properties for full control over which datasets are acted upon. For example, to manage everything under zroot/var except zroot/var/crash and zroot/var/tmp you would do this.
Code: 
% zfs set zap:snap=on zroot/var/
% zfs set zap:snap=off zroot/var/crash zroot/var/tmp
% zap snap -v 3w
Child datasets under zroot/var will inherit zap:snap=on, so you explicitly turn it off for zroot/var/crash and zroot/var/tmp.

For replication, use something like this.
Code: 
% zfs set zap:rep='zap@backup_host:zback/source' zroot/var
% zfs set zap:rep=off zroot/var/crash zroot/var/tmp
% zap rep -Fv
One limitation that I hope to address is the lack of direct support for replication to local datasets. You can set up ssh to localhost and send them with zap that way, but that's clearly not ideal.

ADDED LATER: stratacast1, I now believe we have spoken about all this before. Apologies for the repetition and not directly addressing your original question.
 
I wasn't aware of sysutils/zap. I'll definately have a closer look at it.
Does it support taking snapshots on a remote system?

I'm currently using a modified version of zfsnap which can issue snapshots via ssh on various systems from one central backup server. Replication of datasets is done by a small shellscript on that backup server that takes the ssh-arguments, source and destination dataset and then runs zfs send|receive either with a replication- or incremental stream. Everything is configured within a crontab. I'm still working on a tool to ease manual modification of that rapidly growing crontab though...
The process works fine even from various OSes (FreeBSD, illumos, linux) to the central FreeBSD backup server, and having all backup configuration within a single crontab is also nice; but I'd like to test some ideas for using zfs properties to store all configuration, as this would be the much cleaner way when using ansible for deployment and configuration: just set the properties and let snapshots/replication happen automagically without further configuration.
 
I have one major problem with that: Not all of the remote systems are running FreeBSD. E.g. a majority of snapshots have to be taken for smartOS zones. SmartOS is stateless - so I'd have to pre-seed it via config management with the additional packages. This would work, but is more complex and brings additional maintenance compared to a central tool that just uses the zfs and zpool commands remotely which are avaliable on all ZFS systems.

I wrote down a lot of ideas while building the current solution by just duct-taping together zfsnap2, zfs/zpool, ssh and cron. Maybe one day I will pick up and either write a home-grown tool or (more likely) just further extend zfsnap2 with some more of them. zfsnap is basically also just some glue around zfs and zpool with some added logic, so it is very easily extensible - adding ssh took only a few minutes and 16 lines of new code (to be fair, I could have made it a bit prettier... :oops:). Adding a parser to feed the basic logic of zfsnap with values from zfs properties shouldn't be a problem (zftools does it - so the logic could be borrowed from there). OTOH the currently running solution works very well and cron doesn't seem to care about a crontab with >150 lines; so it still scales "well enough" for me and I suspect cron will be hard to beat performance-wise, especially when reading zfs properties instead...
 
jrm@ said:
sysutils/zap ADDED LATER: stratacast1, I now believe we have spoken about all this before. Apologies for the repetition and not directly addressing your original question.
Click to expand...

Definitely right about talking about zap in the past, and I did see that I could use zap to set that value on each dataset, but I'm primarily concerned with why doing a send/receive is hanging on /usr/ports in this case. I have been playing around with zap already (currently, just doing daily snapshots, will be setting up zap for replication soon enough). My apologies, I blanked on using zap in the context of the issue I brought up in this thread :( I could take SirDice's and phoenix's advice and just ignore /usr/ports in my backup to save space since a portsnap fetch/extract is no big deal, and I just may do that with ZAP. If you do have any insight on why zfs send/receive is hanging on /usr/ports I would be happy to know, in case this happens on another dataset :) I apologize if my string of issues have been addressed before, I try to read the docs and scour the web for answers before I post
 
stratacast1, I didn't connect your username here with the contact information from before.

I haven't experienced hangs with zfs send/receive, so I probably don't have anything useful to add, but here are a few wild guesses.

You have probably done this already, but a first step might be to confirm with certainty that it's hanging. Does zfs send -rnv zroot@`formatting for date` say anything useful? Have you done any system monitoring while the send and receive is hanging? Is the CPU at 100%? What are you seeing with disk i/o? Maybe piping the output to something like textproc/bar could be useful.

Could there be issues with the flash drive? Have you tailed /var/log/messages during the send/recieve to see if any hints are reported?
 
jrm@, I did a zfs send -Rnv before I did the actual send and it came up with no errors at all. I'll have to get back to you on the system information. I did, however, do zpool iostat to see what was going on between my backup pool and zroot, and was sitting at about 50KB/s. However, even after some time had gone by there wasn't any progress and the backup pool had the same remainder of space. The only consistency is it would hang on /usr/ports.

I will be at server-side some time next week to plug in the flash drive and do what you suggest. Fresh FreeBSD box that will be needing its first OS backup next week
 
I am facing a very similar problem using Freebsd11.2. The difference is I am doing this command over the internal network. I found process start to hang when replication is in progress and I do zfs list command. Doing upgrade to 12 and will see if the problem is there too. CPU usage is normal, it just does not show any data is moving when checking on tcpdump. Connections are open. I can kill send process but on receiving machine, it is not possible to to make kill -9 of the process, Restart fixes it.
 
You must log in or register to reply here.
Back
Top