rsync behaviour

N

nbsjbsd

OS: FreeBSD 13.1-RELEASE-p3

I have two 8 TB drives. Drive A is the source drive, Drive B the destination. I do an rsync -av nightly from Drive A to Drive B and it works fine. I don't know why I didn't see this before but after rsync is run, if I manually delete the source file on Drive A, the next day for example, the destination file on Drive B goes away too. The first time I noticed this I thought I was seeing things so I repeated it, making sure I checked the destination drive and sure enough it disappeared.

I'm assuming this is normal behaviour but it is surprising. Any thoughts on how to disable this ?
 
Everything can be found in the man page, rsync(1).
Code: 
       --delete
              This tells rsync to delete extraneous files from the receiving
              side (ones that aren't on the sending side), but only for the
              directories that are being synchronized.  You must have asked
              rsync to send the whole directory (e.g. "dir" or "dir/") without
              using a wildcard for the directory's contents (e.g. "dir/*")
              since the wildcard is expanded by the shell and rsync thus gets
              a request to transfer individual files, not the files' parent
              directory.  Files that are excluded from the transfer are also
              excluded from being deleted unless you use the --delete-excluded
              option or mark the rules as only matching on the sending side
              (see the include/exclude modifiers in the FILTER RULES section).

But, more importantly:
Code: 
       --no-OPTION
              You may turn off one or more implied options by prefixing the
              option name with "no-".  Not all positive options have a negated
              opposite, but a lot do, including those that can be used to
              disable an implied option (e.g.  --no-D, --no-perms) or have
              different defaults in various circumstances (e.g. --no-whole-
              file, --no-blocking-io, --no-dirs).  Every valid negated option
              accepts both the short and the long option name after the "no-"
              prefix (e.g. --no-R is the same as --no-relative).
 
If you don't need to send over the network it's easier to use "clone".
Give it a try.
"-i " works incrementally and never deletes files only adds.
"-s" brings source and destination in sync so deletes files if necessary.

FreshPorts -- sysutils/clone: File tree cloning tool

clone is a file tree cloning tool which runs 3 threads - a scheduler (main), a reader, and a writer thread. Reading and writing occurs in parallel. While this is most beneficial for copying data from one physical disk to another, clone is also very well suited for cloning a file tree to any...
www.freshports.org www.freshports.org

Probably rsync has flags to do similar actions.
 
I use rsync on different versions of FreeBSD, but it never deletes any file with "-av" options.
Please show the complete command line related to rsync, or the whole script.

For troubleshuting you can use "-n" option, it is prevent any modification of filesystem.

Also, try to use options "--backup --backup-dir=" which can save in the separate directory the files which should be deleted on destination.
It is very useful to have previous versions of the files. It works like a cheap time-machine.
Here is an example:
Code: 
#!/bin/sh
date_now=`date "+%Y%m%d-%H%M"`
/usr/local/bin/rsync -Ha -x --delete --numeric-ids --exclude-from=/root/bin/backup-server2tb.exclude.usr --log-file=/var/log/rsync/rsync-server2tb-usr-${date_now} --backup --backup-dir=/home/backup/server.sdm/rsync.backup/usr-${date_now}-backup  /usr/ /home/backup/server.sdm/usr/

Finally, you can make snapshots on the destination (daily/weekly/monthly),
so you will have a possibility to recover any file stored on the destination.
 
im said:
I use rsync on different versions of FreeBSD, but it never deletes any file with "-av" options.
Please show the complete command line related to rsync, or the whole script.

For troubleshuting you can use "-n" option, it is prevent any modification of filesystem.

Also, try to use options "--backup --backup-dir=" which can save in the separate directory the files which should be deleted on destination.
It is very useful to have previous versions of the files. It works like a cheap time-machine.
Here is an example:
Code: 
#!/bin/sh
date_now=`date "+%Y%m%d-%H%M"`
/usr/local/bin/rsync -Ha -x --delete --numeric-ids --exclude-from=/root/bin/backup-server2tb.exclude.usr --log-file=/var/log/rsync/rsync-server2tb-usr-${date_now} --backup --backup-dir=/home/backup/server.sdm/rsync.backup/usr-${date_now}-backup  /usr/ /home/backup/server.sdm/usr/

Finally, you can make snapshots on the destination (daily/weekly/monthly),
so you will have a possibility to recover any file stored on the destination.
Click to expand...
Thanks for the response. It's a simple cron command:

00 03 * * * /usr/local/bin/rsync -av /triglav/4K/ /triglav2/4K > /var/log/rsync_4k.log 2>&1
00 05 * * * /usr/local/bin/rsync -av /triglav/SD/ /triglav2/SD/ > /var/log/rsync_sd.log 2>&1
 
"rsync -av" should not delete any file on the destination.
But check the final slash at the end of the destination of the first line.
In my opinion it should be like this:
00 03 * * * /usr/local/bin/rsync -av /triglav/4K/ /triglav2/4K/

Also show the log records related to the files are deleted on the destination.
 
rsync -auv --delete from/ target

is syncing, updating and delete between two destinations. The slash will copy the directory inside the target. Not what i prefer :)
 
nbsjbsd said:
OS: FreeBSD 13.1-RELEASE-p3

I have two 8 TB drives. Drive A is the source drive, Drive B the destination. I do an rsync -av nightly from Drive A to Drive B and it works fine. I don't know why I didn't see this before but after rsync is run, if I manually delete the source file on Drive A, the next day for example, the destination file on Drive B goes away too. The first time I noticed this I thought I was seeing things so I repeated it, making sure I checked the destination drive and sure enough it disappeared.

I'm assuming this is normal behaviour but it is surprising. Any thoughts on how to disable this ?
Click to expand...

If you don't have --delete that should not happen, period. Unless there are specific reasons such as that file got replaced with a symlink in the source (but then you would find the symlink). In either case, with -v you should get a message about it.

I also recommend `-H --sparse` in the rsync-for-backup commandline to reproduce as much of the source tree as possible.
 
im said:
"rsync -av" should not delete any file on the destination.
But check the final slash at the end of the destination of the first line.
In my opinion it should be like this:
00 03 * * * /usr/local/bin/rsync -av /triglav/4K/ /triglav2/4K/

Also show the log records related to the files are deleted on the destination.
Click to expand...
Good catch. I missed that.

The other entry in cron has that final slash so as a test I deleted a source file and it was not deleted from the destination.

I've added the trailing slash on the other entry and will see what happens tomorrow when I delete a file.
 
Rsync is one of the handiest tools I have ever used, and I use it a lot. Kudos to Tridgell and Mackerras.

However... rsync appears to be configured, by default, for speed, and not correctness. It's dangerous.

It's great for moving huge production datasets of ordinary files, where you can take a week or more for the initial copy with everything on-line, repeat the process as often as you like to catch the deltas, and, on the day of the cut-over, bring down your applications, add "--delete" to the rsync pass, and complete an exact multi-tarabyte copy in minutes.

Like almost everyone else, I have learned the hard way to think about the trailing slashes, and pretty much always use them.

You generally want to preserve sparseness. But there is no specific option to do that (and Unix doesn't help). All you can do is create sparseness, where possible, at the destination. That's good enough in most cases. [I did spend a day once investigating file system shrinkage, to discover some xray images (hidden in many terabytes of other files) that acquired sparseness when copied by rsync.]

You generally want to preserve hard links. Rsync will NOT do this by default.

My standard option set is rsync -SHAXax /path/to/source/ /path/to/destination/ where one of the paths may have an ssh-assisted network spec prepended ("user@host:"). The option set is arduous, but worthy of close examination, at least once.
 
You must log in or register to reply here.
Back
Top