Can't write a simple rsync backup script

[nickednamed]

I'm trying to write a simple net/rsync backup shell script, but I'm having problems with wildcards and escape characters.

There is one line in my script which is causing all the problems, yet it works fine from the console. Essentially, I'm trying to copy my dotfiles and hidden folders from:
home/user/ to home/user/backups/dotfiles.

Here is the command: # rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles

When used in a script, rsync seems to ignore the escape character and will include all my top level home directories, along with /home/user/backups/ in the source directory, which leads to another backup folder in my backup destination, recursively on, forever, i.e. I end up with: /home/user/backups/dotfiles/backups/dotfiles/backups/dotfiles/...

I've tried using quotes, full paths, and many variations of both, but the script will not behave as it does on the command line (many of the variants worked well from the command line). What am I missing here?

 

[DutchDaemon]

Just use rsnapshot, which makes incremental hourly/daily/weekly/monthly backups using rsync, including rollbacks. It takes less time to set up than to write another variant. As to your nested directories, make sure you understand the essential difference to rsync between /some/directory and /some/directory/, especially when using rsync recursively.

 

[wblock@]

sysutils/rsnapshot is not always a replacement for using rsync directly. There are assumptions built in to the way it does things that may not be what the user wants to achieve.

Anyway:

rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.[color="Red"]*[/color] ~/backups/dotfiles

Untested, but the dot is escaped and the star is not. The dot is not a special character to the shell, but it will try to expand the star before running the command. That may be done differently depending on the interactive shell versus the interpreter shell.

 

[NewGuy]

I haven't tested this theory, but I wonder if rsync is respecting symbolic links? I'm assuming your home directory is really under the /usr directory and not under /home. If the script is expanding "~" as /home then rsync might be handling the symlink in unexpected ways. Have you tried putting in the (real) full path to your home directory?

 

[jalla]

Here is the command: # rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles

When used in a script, rsync seems to ignore the escape character and will include all my top level home directories, along with /home/user/backups/ in the source directory, which leads to another backup folder in my backup destination, recursively on, forever, i.e. I end up with: /home/user/backups/dotfiles/backups/dotfiles/backups/dotfiles/...

Your problem is the use of '~/.*' as it matches both the current directory and the parent ('.' and '..'). '.??*' is usually an acceptable workaround.

PS. And I don't believe you when you say this works interactively

 

[phoenix]

I'm trying to write a simple net/rsync backup shell script, but I'm having problems with wildcards and escape characters.

There is one line in my script which is causing all the problems, yet it works fine from the console. Essentially, I'm trying to copy my dotfiles and hidden folders from:
home/user/ to home/user/backups/dotfiles.

Here is the command: # rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles

When used in a script, rsync seems to ignore the escape character and will include all my top level home directories, along with /home/user/backups/ in the source directory, which leads to another backup folder in my backup destination, recursively on, forever, i.e. I end up with: /home/user/backups/dotfiles/backups/dotfiles/backups/dotfiles/...

I've tried using quotes, full paths, and many variations of both, but the script will not behave as it does on the command line (many of the variants worked well from the command line). What am I missing here?

Skip doing it on the command-line, and use an --include-from= file, and list your patterns there.

 

[nickednamed]

Just use rsnapshot, which makes incremental hourly/daily/weekly/monthly backups using rsync, including rollbacks. It takes less time to set up than to write another variant. As to your nested directories, make sure you understand the essential difference to rsync between /some/directory and /some/directory/, especially when using rsync recursively.

Thanks. I had read the part of the manual about trailing slashes, and experimented accordingly. But I'll re-read that part of the manual for good measure.

 

[nickednamed]

Your problem is the use of '~/.*' as it matches both the current directory and the parent ('.' and '..').

'.??*' is usually an acceptable workaround.

PS. And I don't believe you when you say this works interactively

Thanks, I'll try to remember .??* in the future. I'm off to see if it works. And yes, my command does work interactively (just tried it again to make sure! ). Is there some reason it shouldn't other than ~/.* matching the current directory and the parent ('.' and '..')?

 

[nickednamed]

Skip doing it on the command-line, and use an -- include-from= file, and list your patterns there.

Thanks. That's a much cleaner way of doing it! ) I suppose it will be easier to maintain a number of different include files for different purposes too. I'll get cracking on one now!

 

[nickednamed]

OK, so I found a solution (other than the ones already shown to me by others in this thread). I've changed:
# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles

Now it reads:
# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/.[^.]* ~/backups/dotfiles

It works fine now. I'll go back and try a few of the other solutions too. Thanks for the help.

EDIT:

The following also works, both from the command line, and from within the script:
# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/.??* ~/backups/dotfiles

 

[jalla]

Thanks, I'll try to remember .??* in the future. I'm off to see if it works. And yes, my command does work interactively (just tried it again to make sure! ). Is there some reason it shouldn't other than ~/.* matching the current directory and the parent ('.' and '..')?

Stick to your own solution ('.[^.]*'). It's better than my quick-and-dirty workaround.

With regard to the '.*' I have a hard time seeing how this could possibly work. You can do an ls .* to verify that this actually expands to all files both in the current directory and the parent.

 

[nickednamed]

I'm having a few problems with the include and exclude files...

Most of them are working well, but when I try to exclude ALL files except for those in the include list, nothing gets included (I have used full paths throughout).

Here is what I'm trying to do - I've made an include file which lists a bunch of important system files, such as etc/rc.conf and boot/loader.conf which happen to be dotted around the file system, hence the source directory is "/".

Because of this, I want to exclude ALL directories and files, except for the ones listed in the include file:

rsync -avzu --delete --delete-excluded --exclude='*' --include-from=/usr/home/user/.rsync/system.files /* ~/backups/temp

Putting the "exclude" after the "include" has no effect:

rsync -avzu --delete --delete-excluded --include-from=/usr/home/user/.rsync/system.files --exclude='*' /* ~/backups/temp

So how do I exclude all files by default, apart from the ones listed in my include file?

 

[phoenix]

Rsync uses a first-match-wins filter rule set. /* is an implicit include rule for everything under /

Instead of using /* and explicit --include and --exclude files, you should use --filter and create a proper rule list. See the FILTER RULES section of the rsync(8) man page for details. Create a single file that lists the explicit include/exclude rules. And only use / as the source directory on the command-line.