questions about rsync

W

wonslung

Is there an easy way to make rsync copy more than one file at a time?


I have a seedbox set up which will move completed downloads to a dir, and i have set up my local NAS with an ssh key for the seedbox, I can run rsync from my local machine with something like this:

Code: 
rsync -e 'ssh -ax' -avz wonslung@xxxx.xxx.xxx:/home/wonslung/Complete/ /store/Wonslung/seedbox/

but it only copies one file at a time.

Is there any way to increase this to....say, 5 or 10

I can get better bandwidth this way.

on a side note, if i set up a cron job to rsync like this, will it cause a problem if it hasn't FINISHED from the last time?


thanks for the help
 
There is no multithreading in rsync that I am aware of. I remember researching it awhile back while creating a backup solution for my servers. Something about it would require major changes to the protocol.

on a side note, if i set up a cron job to rsync like this, will it cause a problem if it hasn't FINISHED from the last time?
Click to expand...

If the file has been changed from one run of the cron job to the next, whether it be because of an incomplete download or you modified the file, rsync will treat the file as changed and copy the new (entire) file over.
 
well, the cool thing abotu the way i have it set up is...only COMPLETE downloads end up in the folder.


I was more worried about something like this happening...

lets say i set it up to run once an hour, and for some reason a bunch of new files end up in the dir, so many that it starts at 11 oclock and at 12 oclock it's still going...so it starts running again and they both try to update the same file (because all the ones before that would be done)

would this cause a problem?


anyways, for now i just set up my script like this:


Code: 
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
LPATH="/store/Wonslung/seedbox/"
RPATH="wonslung@xxx.xxx:/home/wonslung/Complete/"
LOGFILE=/home/wonslung/log/backup

if pgrep -u $USER rsync
then 
	echo "rsync is running"
else
rsync -e 'ssh -ax' -av --delete $RPATH $LPATH >> $LOGFILE
fi

that way if rsync is already running under my user, it just skips downloading until the next cron time.

If it's not a problem then i'll alter it.

actually, with the way it is, i might just set it to run every 30 minutes......that wouldn't cause any real errors would it?

edit:

also, i've noticed that rsync seems to open 3 processes when i run it this way...is that normal?
Code: 
wonslung 11122  0.0  0.0 20092  1944  ??  S     4:00PM   0:00.01 rsync -e ssh -ax -av --delete wonslung@xxxxxx.xxxxx.xxx:/home/wonslung/Complete/ /store/Wonslung/seedbox/
wonslung 11123  0.0  0.1 16380  6636  ??  S     4:00PM   0:19.66 ssh -ax -l wonslung xxxxxx.xxxxx.xxx rsync --server --sender -vlogDtpre.iL . /home/wonslung/Complete/
wonslung 11152  0.0  0.0 21116  2212  ??  S     4:00PM   0:06.09 rsync -e ssh -ax -av --delete wonslung@xxxxxx.xxxxx.xxx:/home/wonslung/Complete/ /store/Wonslung/seedbox/

i understand why it might open 2....but out of those 3, 2 look identical....
 
That looks fine, I would have added in a lock file and a test expression for that, the downside there is if the script terminates abnormally and doesn't delete the lock in the end, your cron wouldn't run (afterthought, guess you could append a timestamp on the end of the lockfile name or something).

As for the processes, I know you should have at least two, rsync and sshd, I can't say for certain if the third is normal or not.
 
I'm still very green when it comes to shell scripting. I'll read up on lock files and such, thanks for the suggestion...yah, i'm not sure why it's got 3 going....i don't know, perhaps i did something wrong...
 
I don't know the exact purposes of each process, but I always get three processes for each rsync connection: 1 sshd, 2 rsync. Seems to be normal.
 
maybe one is the rsync "daemon" which has an rsync worker on either side of it...or something

who knows.
 
You must log in or register to reply here.
Back
Top