Solved Parsing rsync command options - how to quote

jontheil

jontheil

Hi forum,
I am trying to create a simple script to make incremental backups of my FreeBSD server into a FreeNAS box by using rsync(1) and some programme options.
The goal is to make a script that can run from cron(8) on a daily basis.
Before I can reach the goal, the basic functionality must be right.
For the moment, the script looks like this:
Code: 
#!/bin/sh
SRC=" / "
DST="root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/"
LNKDST="--link-dest=../full"
FULL="/full"
EXCL="--exclude '/home' --exclude 'dev/' --exclude '/storage' --exclude '/mnt' "
SSHCONNECT="-e    'ssh -p 9851'"
DATE="daily-`date +%Y-%m-%d`"
OPT="-az --ignore-existing --stats "
LOG=" --remote-option=--log-file=/mnt/naspool/nasstorage/home/my-home-dir/rsync.log "
rsync ${OPT}${SSHCONNECT}${LOG}${EXCL}${LNKDST}${SRC}${DST}${DATE}
The output shows an error:
Code: 
# ./root/bin/rsync_incremental.sh
Missing trailing-' in remote-shell command.
I know this has to do with variable expansion in the shell. I have searched the Internet to look for solutions. I can see this is a common problem, and that the solution might be to use arrays in the script. I just cannot figure out how to do that.

Any help is very much appreciated.

Regards,
Jon
 
The shell hates quotes of all kinds and will mess with them at every opportunity.

The exclude string does not really need any of those single quotes, none of the directories have spaces or special characters.

The parameter for the final command at the bottom should probably be quoted to prevent the shell from "helping" interpret whitespace and quotes:
Code: 
rsync "${OPT}${SSHCONNECT}${LOG}${EXCL}${LNKDST}${SRC}${DST}${DATE}"

FreeBSD's sh(1) does not support arrays, although I don't see how they would help here.
 
Thanks for your reply.
It seems that the double-quoting of the command helped parsing the remote shell command correctly. It still does not work, though.
The output is a very log line, so I have put the exclusion rules into a separate file with the --exclude-from switch.
I will try to elaborate the script more. And if I do not find a solution, I will make a new post.
 
Don't mix single and double quotes. Use double quotes everywhere, and just escape (\") the internal quote marks.
Code: 
SSHCONNECT="-e \"ssh -p 1234\""

Or, remove all the internal quote marks from the variables, and put them into the command line:
Code: 
SSHPORT="1234"
rsync ${OPT} -e "ssh -p ${SSHPORT}" ...

And don't embed spaces into the variables. Instead, put spaces between the variables in the command line, along with / for paths, etc.. For example, which of these is easier to understand:
Code: 
rsync ${OPT}${SSHCONNECT}${LOG}${EXCL}${LNKDST}${SRC}${DST}${DATE}

rsync ${OPT} -e "ssh -p ${SSHPORT}" ${LOG} ${EXCL} ${LNKDST} ${SRC} ${DSTUSER}@${DSTHOST}:/${DST}/${DATE}/
 
phoenix said:
Don't mix single and double quotes. Use double quotes everywhere, and just escape (\") the internal quote marks.
Click to expand...
Sometimes single quotes are required to prevent interpolation.

phoenix said:
rsync ${OPT} -e "ssh -p ${SSHPORT}" ...
Click to expand...
Yes, but escapes or quotes or both are often necessary to prevent sh(1) from "helping":
Code: 
% cat tester.sh
#!/bin/sh
SSHPORT="1234"
echo rsync -e "ssh -p ${SSHPORT}"
echo rsync -e \"ssh -p ${SSHPORT}\"
% ./tester.sh
rsync -e ssh -p 1234
rsync -e "ssh -p 1234"

If the programs are non-trivial, it can save much time and aggravation to use a more modern language like Perl, Python, or Ruby.
 
BTW, --exclude statements are generally written like
Code: 
--exclude=home/
etc. The = does belong there.
 
From a script that I use to rsync OpenBSD snapshots from URL's like rsync://ftp.nluug.nl/openbsd/ :
Code: 
skip="
cdboot
cdbr
game57.tgz
install57.fs
install57.iso
miniroot57.fs
"
for THIS in ${skip} ; do
  EXCLUDE="${EXCLUDE} --exclude='${THIS}'"
done

echo ${EXCLUDE}
echo
echo ==========================================

#CMD="sudo rsync -tr -c -vv --progress --stats --dry-run --itemize-changes ${MIRROR}${SRCDIR} ${DESTDIR}"
CMD="sudo rsync -tr -vv --progress --stats --itemize-changes ${EXCLUDE} --delete-excluded ${MIRROR}${SRCDIR} ${DESTDIR}"

echo $CMD
printf "Press a key to continue or press CNTRL-C to abort ...." ; read A
eval ${CMD}
As you can see I construct a command in a shell variable, show it and then execute it with eval, although in this case this is not really required. The advantage is that is can visually inspect the command before actually running it.

Code: 
[cmd=$]snap-rsync.sh amd64 nluug[/cmd]

--exclude='cdboot' --exclude='cdbr' --exclude='game57.tgz' --exclude='install57.fs' --exclude='install57.iso' --exclude='miniroot57.fs'

==========================================
sudo rsync -tr -vv --progress --stats --itemize-changes --exclude='cdboot' --exclude='cdbr' --exclude='game57.tgz' --exclude='install57.fs' --exclude='install57.iso' --exclude='miniroot57.fs' --delete-excluded rsync://ftp.nluug.nl/openbsd/snapshots/amd64 /home/www/snapshots/

Press a key to continue or press CNTRL-C to abort ....^C
[cmd=$][/cmd]
 
J65nko said:
As you can see I construct a command in a shell variable, show it and then execute it with eval
Click to expand...

Never use eval!

The way you build the EXCLUDE variable is problematic and should be reconsidered. Depending on how long rsync runs and how long sudo keeps the password you're in trouble if someone gains control over your environ(7)ment. And no, echo won't help you...

Edit: Scratch the "depending on" part above if it's possible to make rsync fail silently (like, by setting EXCLUDE to "--x 2>/dev/null; : ...nasty stuff here...").

Code: 
% cat badidea.sh
#! /bin/sh
for i in 3; do ARGS="${ARGS} $i"; done
CMD="sudo sleep $ARGS"

echo "CMD=$CMD"
read -p "Ok? " key

eval "$CMD"
%
Code: 
% sh badidea.sh
CMD=sudo sleep  3
Ok?
Password:
%
Code: 
% export ARGS="2>/dev/null;: $(printf '\b\b\b\b\b\b\b\b\b\b\b\b\b\b\0337'); echo fun with eval, sudo and vt100 escape sequences; sudo id; sleep 3; : $(printf '\0338\033[0K')"
% sh badidea.sh
CMD=sudo sleep  3
Ok?
Password:
fun with eval, sudo and vt100 escape sequences
uid=0(root) gid=0(wheel) groups=0(wheel),5(operator)
%
 
I tried to follow some of the advices, but still can't make it work.
Short version of the script:
Code: 
#!/bin/sh
SRC=" / "
DST="root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/"
LNKDST="--link-dest=../full"
SSHCONNECT="-e \'ssh -p 9851\'"
DATE="daily-`date +%Y-%m-%d`"
OPT="-az --ignore-existing --stats --exclude-from=/root/rsync/excl "
LOG=" --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log "
rsync "${OPT}${SSHCONNECT}${LOG}${LNKDST}${SRC}${DST}${DATE}"
When I run it with sh -vx, I get an output of both the variables and the command itself:
Code: 
# sh -xv /root/bin/rsync_incremental.sh
SRC=" / "
+ SRC=' / '
DST="root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/"
+ DST=root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/
LNKDST="--link-dest=../full"
+ LNKDST=--link-dest=../full
SSHCONNECT="-e \'ssh -p 9851\'"
+ SSHCONNECT='-e \'\''ssh -p 9851\'\'
DATE="daily-`date +%Y-%m-%d`"
+ date +%Y-%m-%d
+ DATE=daily-2015-01-13
OPT="-az --ignore-existing --stats --exclude-from=/root/rsync/excl "
+ OPT='-az --ignore-existing --stats --exclude-from=/root/rsync/excl '
LOG=" --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log "
+ LOG=' --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log '
rsync "${OPT}${SSHCONNECT}${LOG}${LNKDST}${SRC}${DST}${DATE}"+ rsync '-az --ignore-existing --stats --exclude-from=/root/rsync/excl -e \'\''ssh -p 9851\'\'' --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --link-dest=../full / root@mfl-nas:/mnt/naspool/nasstorage/backup/mflserver4/rsync/daily-2015-01-13'
rsync: -az --ignore-existing --stats --exclude-from=/root/rsync/excl -e \'ssh -p 9851\' --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --linrsync error: syntax or usage error.
Most of it seems reasonable. But somehow the --link-dest option is truncated. Any explanation for that?
 
I don't know about "--link-dest" truncation (maybe a limit of sh(1) ?), but the error could be caused by this (see rsync(1)):
Note some versions of the popt option-parsing library have a bug
in them that prevents you from using an adjacent arg with an
equal in it next to a short option letter (e.g.
-M--log-file=/tmp/foo. If this bug affects your version of
popt, you can use the version of popt that is included with
rsync.
Click to expand...
 
Update:
It can have something to do with some setting in my shell. I have seen strange truncations at other occasions. I am running the thing from a Putty terminal, but I don't think it is related to that fact. But maybe some system setting regarding the shells behaviour. I am using csh(1).
 
Try to factor the cumbersome -e option into an environment variable. Something like this:

Code: 
env RSYNC_RSH="ssh -p 9851" rsync "${OPT} ...

I wouldn't encode the spaces directly into the variables but place them on the command line where appropriate.

Code: 
SRC="/"

And then:

Code: 
rsync "... ${LOG} ${LNKDST} ${SRC} ${DST}${DATE}

Also do the same with other options, rearrange them so that the values don't have spaces in them.

Your shell has no effect because you're telling the system to use /bin/sh as the interpreter on the first line of your script.
 
Thanks for the suggestion.
I have changed the script a bit trying to implement the suggested changes:
Code: 
SRC="/"
DST="root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/"
LNKDST="--link-dest=../full"
DATE="daily-`date +%Y-%m-%d`"
OPT="-az --ignore-existing --stats --exclude-from=/root/rsync/excl"
LOG=" --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log"
env RSYNC_RSH="ssh -p 9851" rsync "${OPT}${LOG} ${LNKDST}${SRC} ${DST}${DATE}"
I still get an error. To me, it looks like some kind of truncation. Though now it seems more strange, since a don't refer to anything like "roo" (or root) after ${LNKDST}.
# /root/bin/rsync_incremental.sh
rsync: -az --ignore-existing --stats --exclude-from=/root/rsync/excl --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --link-dest=../full/ roorsync error: syntax or usage error
 
Correction: I missed a space after ${LNKDST}, and the the next argument is of course root@mfl-nas.
 
If you put the spaces in the final command rather than the variables, that would be easier to see and debug.
 
I don't understand what you mean. I have put spaces around the variables in the final command. And I have removed the trailing spaces from the variable definitions. But I guess you mean something else than that. Of course I could omit all the variables - except maybe for the date variable, but then it would be a very long command line and not really a script. :)
 
There is still a leading space in ${LOG}. Remove that, and make the final command:
Code: 
rsync "${OPT} ${LOG} ${LNKDST} ${SRC} ${DST} ${DATE}"
 
Made the suggested corrections literally.
But still, I get the error:
Code: 
# sh -x /root/bin/rsync_incremental.sh
rsync: -az --ignore-existing --stats --exclude-from=/root/rsync/excl --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --link-dest=../full / rorsync error: syntax or usage error (code 1) at main.c(1575) [client=3.1.1]
My knowledge of shell scripting is extremely limited. Just trying to learn. I don't know if I am making a fundamental error or if /bin/sh has some limitation in this case. There still seems to be a truncation at the end of the main command. I don't get it.
Thanks for trying to help!
 
Where is that extra "ro" or "roo" in the error message coming from?

In any case, basic debugging: use echo to display the values of variables before the final command is called. Insert an echo before that command to show the line it will use*, rather than run it.

(*: As always, it might helpfully remove some quotes.)
 
jontheil said:
Made the suggested corrections literally.
But still, I get the error:
Code: 
# sh -x /root/bin/rsync_incremental.sh
rsync: -az --ignore-existing --stats --exclude-from=/root/rsync/excl --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --link-dest=../full / rorsync error: syntax or usage error (code 1) at main.c(1575) [client=3.1.1]
Click to expand...
I'm not too advanced with sh(1) yet, but I vaguely remember a post on one of the mailing lists a few months back that talked about an error similar to this. I think it had something to do with a port option that uses an external library for compression or something similar. I can't remember off hand. If that happens to be the case here, it may be net/rsync itself at fault here and not the script.
 
Okay, here is what I tried. With some answers and still a question.
I put back the SSHCONNECT variable (without backslashes) and tried to put echo instead of rsync:
Code: 
echo "${OPT} ${SSHCONNECT} ${LOG} ${LNKDST} ${SRC} ${DST}${DATE}"
The output seemed to look as intended:
# /root/bin/rsync_incremental.sh
-az --ignore-existing --stats --exclude-from=/root/rsync/excl -e 'ssh -p 9851' --remote-option=--log-file=/mnt/naspool/nasstorage/home/home-dir/rsync.log --log-file=/root/rsync/rsync.log --link-dest=../full / root@mfl-nas:/mnt/naspool/nasstorage/backup/server/rsync/daily-2015-01-14
When I grapped the output and put rsync in front, the command worked just fine.
Still, knowing almost nothing about shell scripting, I suppose /bin/sh somehow is creating the problems. If that is the case, I don't know how to solve it. Using another shell maybe? Or trying to find out which part of the current script is causing the trouble?
 
The next step would be to remove the quotes around the parameters, because that might be giving rsync just one parameter:
Code: 
rsync ${OPT} ${LOG} ${LNKDST} ${SRC} ${DST} ${DATE}
 
Got the "original" error message in first try. Removed {SSHCONNECT} again and put it into an env statement in front of rsync once more. Seems to work now. I'll report back when a see the result. Glad already. :)
 
You must log in or register to reply here.
Back
Top