help with rsync/unrar script

wonslung · Mar 5, 2010

changing the pipes to && made it work.

DutchDaemon · Mar 5, 2010

Separate them with ";" or "&&" perhaps? I don't understand all the pipes.

Or replace the quoted variables with ${SRC} and ${SRC2}.

wonslung · Mar 5, 2010

ok, this is the finished (for now, until i can make unrar work) script

Code:

#! /bin/bash
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

next thing i need to do is make it unrar files to a new dir, and delete the rar files/dirs

Thanks so much DD, your help has been invaluable.

wonslung · Mar 7, 2010

so now i need to add the unrar part of the script. currently, this script is doing pretty much what i want when it comes to downloading....so heres what i want to do next.

I need to be able to recursively unrar stuff, and dump what comes out to a single directory. I understand how to do this, but where i run into problems is when i end up with split rar files all named .rar so what i need to do is somehow ignore all but one of those OR make unrar skip unraring if the file already exists....i'd RATHER do the first option if possible...

Also, i'd like to be able to delete the rar files when they are done being extracted, and all split rars' with them...basically, the entire dir they are in..but i don't want to accidently delete stuff that hasn't been processed....so....this is going to be harder than the other part....

When i'm done though, i should have a script which will auto-download everything (via cron) from my seedbox, unrar it all into a dir which can be processed for sorting later.

wonslung · Mar 7, 2010

ok, so i figured out how to find rar files and unrar them into thier dir's...what i want to do now is, if the unrar is successful (how do i tell this?) i want to delete all the .rar and .r[\d\d] files and move the directory containing the unrarred file.

The problem i'm having is how do i know if it's successful.....anyways, this is what i have so far..

Code:

for d in `$FIND	"$DST"*	-type d`
   do
      for r in `$FIND $d -name "*.rar*"`
	 do
            unrar e -r -inul $r $d
      done   
done

the $FIND part is set to my find command. $DST is the same from earlier in the thread.

thanks

wonslung · Mar 7, 2010

i'm wondring....this seems to work:

Code:

for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done

edit 2

i'm too tired...i see what i did wrong....fixed it

(i forgot to add the FIND= line"

does anyone see any issues with this?

edit:

ok, this is odd...when i put the two scripts together i get this error:

Code:

./seedbox.sh: line 23: /tank/nas/dump/torrents/seedbox/Caprica.S01E06.720p.HDTV.x264-CTU: cannot execute [Is a directory]

but they work fine in two separate scripts....why is this?

here is the entire script which isn't working

Code:

#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done


rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

wonslung · Mar 7, 2010

ok, i figured it out, i forgot to set the FIND vairable.....really dumb...anyways, here is my completed script which seems to work....if anyone sees any obvious issues let me know.

Code:

#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done


rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

wonslung · Mar 7, 2010

ok, i have a problem.....this script fails on files with spaces in the names.....any idea how to fix that?

sixtydoses · Mar 7, 2010

Try to add this in your script.

Code:

IFS='
'

wonslung · Mar 7, 2010

sixtydoses said:
Try to add this in your script.

Code:

IFS=' '

i have no idea what this means....

edit:

Just read up on IFS, will try.
not sure where to add that to though

sixtydoses · Mar 7, 2010

Add it before you do the copying. Sorry, I can't make your script to work, but I've faced this white space problem before.

Example:

Code:

[od@meh ~/temp]$ ls -l
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:51 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:52 white space dir

[od@meh ~/temp]$ srcf=`ls`
[od@meh ~/temp]$ echo $srcf
white space white space dir

[od@meh ~/temp]$ cp -vR $srcf ../dst/
cp: dir: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory

[od@meh ~/temp]$ IFS='
> '

[od@meh ~/temp]$ cp -vR $srcf ../dst/
white space dir -> ../dst/white space dir
white space -> ../dst/white space

[od@meh ~/temp]$ ls -l  ../dst
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:55 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:55 white space dir

wonslung · Mar 7, 2010

sixtydoses said:

Add it before you do the copying. Sorry, I can't make your script to work, but I've faced this white space problem before.

Example:

Code:

[od@meh ~/temp]$ ls -l
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:51 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:52 white space dir

[od@meh ~/temp]$ srcf=`ls`
[od@meh ~/temp]$ echo $srcf
white space white space dir

[od@meh ~/temp]$ cp -vR $srcf ../dst/
cp: dir: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory

[od@meh ~/temp]$ IFS='
> '

[od@meh ~/temp]$ cp -vR $srcf ../dst/
white space dir -> ../dst/white space dir
white space -> ../dst/white space

[od@meh ~/temp]$ ls -l  ../dst
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:55 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:55 white space dir

adding that works for spaces but not " breaks it.

Still, this is progress

DutchDaemon · Mar 7, 2010

Use double quotes around variables to include whitespace literally.

So not $FILE, but "${FILE}". The curly braces prevent interaction with adjacent characters which may make the script not recognise a variable when it's called (e.g. PATH=/usr/ ; cd $PATHsrc -- this should be ${PATH}src)

wonslung · Mar 8, 2010

DutchDaemon said:
Use double quotes around variables to include whitespace literally.

So not $FILE, but "${FILE}". The curly braces prevent interaction with adjacent characters which may make the script not recognise a variable when it's called (e.g. PATH=/usr/ ; cd $PATHsrc -- this should be ${PATH}src)

I've been reading up on this issue and it seems to be caused by the ` ` quotes....i have no idea how to fix this issue.

I will try your modifications too but see this: and tell me what you think.

http://mywiki.wooledge.org/BashPitfalls#for_i_in_.60ls_.2A.mp3.60

I have no idea how to rewrite my script without this.

edit: I made some modifications....the whitespace thing is solved but it's solved by a method which breaks other stuff....the problem is two fold.

First, ssh and scp have a whitespace problem ....i had to change this:

Code:

scp -r ${RMT}:$f ${DST}

to this:

Code:

scp -r ${RMT}:"\"$f\"" ${DST}

to make it get paste the whitespace issue....but that breaks stuff with "'s in it.....i think the ultimate issue is that using the `'s are bad.
I tried to fix that by using this: $( ) but it didn't help much

anyways, this is what i have so far but i think i'm going to have to re-write everything to get it to work flawlessly....luckily it works for 99% of my files...so i have itme

Code:

#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in $(ssh $RMT "find ${SRC}* -type d -newer $TMST && \
        find ${SRC}* -type f -maxdepth 0 -newer $TMST && \
        find ${SRC2}* -type d -newer $TMST && \
        find ${SRC2}* -type f -maxdepth 0 -newer $TMST")
     do
 scp -r ${RMT}:"\"$f\"" ${DST}
done

for d in $($FIND ${DST}* -type d)
   do
      for r in $($FIND $d -name "*.rar*")
         do
            unrar e -r  $r $d;rm ${d}/*.r??; mv ${d} ${DST}Done
      done   
done

for m in $($FIND ${DST} -maxdepth 1 -name "*.avi" -o -name "*.mkv")
   do
       mv $m ${DST}Done
done
rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

sixtydoses · Mar 8, 2010

It is a good practice to define IFS when you need it, and reset it back when you're done with it.

e.g.:

Code:

# store original IFS and assign a different value to it
OLDIFS=$IFS
IFS='
'
 
# do whatever you want here
 
# reset it back
IFS=$OLDIFS

wonslung · Mar 8, 2010

cool, doesn't help me with this issue though.

But, i do like learning about "best practices" apparently my script is FULL of "horrid" issues...

sixtydoses · Mar 8, 2010

wonslung said:
to make it get paste the whitespace issue....but that breaks stuff with "'s in it.....i think the ultimate issue is that using the `'s are bad.
I tried to fix that by using this: $( ) but it didn't help much

Sorry I don't really understand.. you were saying you're having problems transfering files "'s" in it? Something like, "some movie's.rar"?

wonslung · Mar 8, 2010

sixtydoses said:
Sorry I don't really understand.. you were saying you're having problems transfering files "'s" in it? Something like, "some movie's.rar"?

actually, i think single quotes (apostrpe's) work..it's double quotes which break it. Theres soemthing else which breaks it as well but it's pretty minor.

one of the biggest problems is that scp itself doesn't handle whitespace, so to make the whitespace work, i have to do this:

scp -r remoteserver:"\"file name with whilespace\"" Destination

This fix breaks sending quotes i think...

LiMPiNg · Mar 12, 2010

First, I would like to thank you wonslung for all the work you have put in this script already. I was looking for something similar to use on my FreeNAS server, without the copying from the seedbox - I just want it to automatically extract files from my download directory to another directory of extracted files. Being a FreeBSD/UNIX rookie, I stumbled my way through it and have it working how I want... almost!

What do you do for archives that have .partXX.rar filenames instead of .rXX? ie, xxxx.part01.rar, xxxx.part02.rar . I ran into such a torrent, which is not uncommon, and it threw the whole script off due to all the .rar files in the dir. I would love to figure this out on my own, but I know nothing about scripting and simply modified yours to fit my needs.

I know the following code is what needs to be worked on for this.

Code:

for r in $($FIND $d -name "*.rar*")

I am guessing having it do an if search for a part01.rar and extract that if it exists and then do an else to process the single .rar if there is no part01.rar is the key, but how to implement that exactly is beyond my limited scripting abilities. Any help would be appreciated.

wonslung · Mar 12, 2010

LiMPiNg said:
First, I would like to thank you wonslung for all the work you have put in this script already. I was looking for something similar to use on my FreeNAS server, without the copying from the seedbox - I just want it to automatically extract files from my download directory to another directory of extracted files. Being a FreeBSD/UNIX rookie, I stumbled my way through it and have it working how I want... almost!

What do you do for archives that have .partXX.rar filenames instead of .rXX? ie, xxxx.part01.rar, xxxx.part02.rar . I ran into such a torrent, which is not uncommon, and it threw the whole script off due to all the .rar files in the dir. I would love to figure this out on my own, but I know nothing about scripting and simply modified yours to fit my needs.

I know the following code is what needs to be worked on for this.

Code:

for r in $($FIND $d -name "*.rar*")

I am guessing having it do an if search for a part01.rar and extract that if it exists and then do an else to process the single .rar if there is no part01.rar is the key, but how to implement that exactly is beyond my limited scripting abilities. Any help would be appreciated.

dude, i'm pretty new to this too and this script is REALLY bad.

i'm also thinking of changing how i do unrar.....i'm thinking instead of using a "for" loop, i'll just use something more like:

find somedir -type f -name *.rar -exec unrar e {} \;

but...that's not quite right and needs some work....i'm very new to this too..

I've been thinking a lot about it and i'm actually thinking about changing my method....this is what i've been thinking of, if anyone has any ideas how i can do this let me know because i'm just in the "idea" phase.

The Problem:

As i've said before, this script breaks on specific file names. I've been thinking about it and rsync is just a better tool, but the major issue is that it doesn't work for what i want and writing a script for an exclude file just isn't easy at all....so this is my idea:

somehow use symbolic links.

Write a script to create symbolic links (should be easy with the same method i used for downloading)

Then have rsync download this directory...the problem is i need to then have rsync DELETE the link when it's done.

any ideas?

edit:

another big problem i think i forgot to mention is:

on very large files, it messes up.

situation: downloading a giant file, it takes some time, meanwhile some new stuff is added, the script doesn't pick it up.

wonslung · Mar 12, 2010

This is what i'm about to test.

If this works the way i THINK it will, i'll be ok....i haven't put the unrar part in let, but i will

The reason i run the rsync command 3 times is i want to make sure i catch everything...tha'ts probably not foolproof....but i don't know what else to do....any ideas would be helpful.
nayways, this is what i'm ABOUT to test...

Code:

#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox2/
TMST=$SRC.timestamp
LOG=/export/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG
rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG
rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG


ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;
rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

wonslung · Mar 12, 2010

ok, i had an error in it, but this is the script which is working:

this goes in, finds dirs which are new, makes a link for them in another dir, then rsync (following links) those dir

I set it to rsync 3 times, hoping that if something is added while the first rsync is going on, it will catch it....i'm trying to think of a better method for this..i'm thinking perhaps a double or tripple timestamp but i'm not entirely sure...must be a better way.

anyways, after that, it goes in and deletes the symbolic links.

heres the script, later i will add the logic to unrar and sort stuff

Code:

#! /bin/sh
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
LOG=/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG
rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG
rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

EDIT:

I just realized the 3 rsync is stupid because it doesn't find any new files, i wasn't thinking....

so my question is: other than just making it run everything again, is there any way to tell it to loop a couple times?

EDIT:

just found the answer. a function.

heres editted script:

Code:

#! /bin/sh
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
LOG=/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

link() {

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"
}

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

Edit again:

okay, i'm SURE this has it's own issues which i'm not seeing yet.....but this is the best i can do so far.

Code:

#! /bin/bash
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
LOG=/home/wonslung/log/rsync.log
IFS='
'

if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

link() {

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"
}

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG


shopt -s extglob

for dir in ${DST}*/
  do
    cd "$dir" && $FIND . -type f -name '*.rar' \( ! -name 'part*.rar' -o -name 'part01.rar' \) -exec unrar e {} \;
done
for g in ${DST}*/
  do
    rm ${g}*.r?? ${g}*.sfv ${g}*.nfo
done
$FIND ${DST}* -maxdepth 0 -type d -exec mv {} /tank/nas/dump/Done \;




ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"

help with rsync/unrar script

wonslung

DutchDaemon

Administrator

wonslung

wonslung

wonslung

wonslung

wonslung

wonslung

sixtydoses

wonslung

sixtydoses

wonslung

DutchDaemon

Administrator

wonslung

sixtydoses

wonslung

sixtydoses

wonslung

LiMPiNg

wonslung

wonslung

wonslung