FTP cronjob : Upload from queue until full?

S

Sokonomi

Hello all,

I, as a total freeBSD beginner, will probably be completely in over my head for this one, but please try anyway. :p

What I want to achieve;
The Laymen's explanation; I want my FreeNAS to upload new TV shows to my tablet every night, until the tablet is full.

I need a script to attempt uploading the top line of a queue file to a FTP destination and delete the top line if it succeeds, cycling until it gets a failure and then quitting. I would run it through a cronjob on my NAS.

Supposedly I have my TV show fetchers generate a list of new shows that looks like this;

/mnt/Tank1/TVshows/Battlebots s3e2.mkv
/mnt/Tank1/TVshows/Battlebots s3e3.mkv
/mnt/Tank1/TVshows/Battlebots s3e4.mkv

Can I make a script that works these into an FTP transfer?

Ive googled around and found an app called C-Kermit, supposedly this makes it easier?
If someone knows of an example script that handles queued FTP uploads in freeBSD I can dissect it and learn how it works. :-/

I hope you guys can help. If its too difficult ill have to resort to running DOS in a VM to do it with a batch file or something. O:‑)

Cheers,
- Soko
 
Yes, great, except this would run in a jail which is basically a flat freeBSD, is it not?
Plus I assumed ftp() is a base freeBSD function.
 
Sokonomi said:
Yes, great, except this would run in a jail which is basically a flat freeBSD, is it not?
Click to expand...
That's really not the point, it would have helped if you had actually read that URL; it isn't just your usual "This is offtopic, please cease" kind of answer. It is also about protection; both your system and our invested time.

I only know FreeBSD so I have no idea what changes have been made to FreeNAS. Don't forget: everything is customizable, so for all I know some binaries which are part of FreeBSD could have been removed on FreeNAS (to save space, or because it's not useful, etc, etc.). But it's also very likely that some things behave in a completely different way; making it very possible that certain answers to problems would easily work on FreeBSD but could actually do massive damage on FreeNAS (I'm not joking nor trying to make things look more gruesome just to make a point).

Another problem is that some derivatives are even based on unstable FreeBSD releases (developer snapshots) which use is actually discouraged by the FreeBSD community itself.

Most importantly: please note that SirDice never locked the thread nor forbid us from discussing.

Sokonomi said:
Plus I assumed ftp() is a base freeBSD function.
Click to expand...
There is no such thing as ftp() (as a function):
Code: 
peter@zefiris:/home/peter $ man 4 ftp
No manual entry for ftp
But there is something such as ftp(1), which I think is exactly what you need. If you read the manualpage you'll notice the -u option which allows you to upload file(s) to a remote destination.

So... maybe something like: ftp -u upload@192.168.0.20 * and then let it just sent everything over, it will complain (and quit) on its own once the remote no longer has any storage space.

To make this run every day you'd want crontab(1). First make a script which performs the upload:

Code: 
#!/bin/sh
cd /mnt/tank1/TVShows
ftp -u upload@192.168.0.20 *
Then add a crontab entry which runs this script every day: crontab -e, for the right syntax to specify the time check crontab(5). Just be sure to specify the full name of your script. So, for example: /home/me/bin/upload-shows, crontab has a limited PATH.

This should be enough to get you started.
 
ftp(1) is just the actual transfer part though, which is easy enough and well documented.
What I need help with is processing the queue list, that stuff needs some hoop jumping I think.
Can you make a script read ftp exit codes?
C-Kermit apparently gives you a little more control over it, but im not sure how to handle the queue part.

The shows have been coming in faster than I can view them, and my tablets storage is limited,
so im having to deal with a bit of a backlog, thats why I cant just dump the entirety of /mnt/tank1/TVshows across.
If I can make a script FTP the files in like a feed, that would be most ideal.
 
Sokonomi said:
What I need help with is processing the queue list, that stuff needs some hoop jumping I think.
Can you make a script read ftp exit codes?
Click to expand...
Ah yes, that's the part I somewhat overlooked.

Sokonomi said:
Supposedly I have my TV show fetchers generate a list of new shows that looks like this;

/mnt/Tank1/TVshows/Battlebots s3e2.mkv
/mnt/Tank1/TVshows/Battlebots s3e3.mkv
/mnt/Tank1/TVshows/Battlebots s3e4.mkv

Can I make a script that works these into an FTP transfer?
Click to expand...
I don't know about you, but I know I can ;) Sorry for the wee bit of pun, weekend started and I'm in a good mood right now.

First: we're not going to be using any 3rd party stuff. The base system provides us with all the tools we need to make this happen. Second, because I'm also currently working on another hobby project of mine I decided to combine the two. So, what you're looking for based on functionality is roughly something like this:

1532716199836.png


For the record: Yes, I am geekish enough to enjoy UML. And no, I'm not trying to show off or anything; it's a hundred degrees out here which doesn't help with my ability to concentrate. As such a quick design like this seriously helps me focus my thoughts. And since I had my design made anyway I figured I'd upload it to show off (but secretly!) :cool:

Anyway, enough fun talk:

Code: 
#!/bin/sh

# Queue file (use full path!)
queue=files.txt
# FTP host
host=breve
# FTP username
user=anonymous
# FTP password
pass=password
# FTP upload path
dir=/upload

## Don't change stuff beyond this line

count=0
size=`wc -l $queue | cut -d ' ' -f8`

for a in `cat $queue`; do
        check=$count;
        ftp -u ftp://$user:$pass@$host$dir/`basename $a` $a && count=$((count+1));
        if [ $check -eq $count ]; then
                echo "There was an error during the upload."    > /dev/stderr;
                break;
        fi
done

newsize=$((size - count))
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue
Warning: I expect you to test this for yourself as well before usage! I did some proper testing and this works on my end, but as mentioned before I have 0 experience with FreeNas.

So... I think this speaks for itself. If you want to upload to the root directory then just leave dir empty. That should work but I haven't tested that. If it doesn't just let me know. Actually.. I'd appreciate it if you'd let me know anyway if this worked or not :)
 
Holy cow, I was expecting some examples so I could piece something together,
I didn't expect someone to do the whole thing for me. :eek: Thanks!

That UML diagram is fantastic, I should have a look at that myself sometime.
Seems to be a great visual help for tracing how your code is gonna need to go.

As it appears, freeBSD script coding looks eerily familiar with other types of coding ive tangled with,
I can almost fully understand whats going on even though I've never written for freeBSD before.

So let me see if I interpreted it right;
It sets 'size' to 'queue' line count.
It runs a loop for every line in 'queue'.
It sets 'check' to 'count'.
It adds +1 to 'count' unless ftp encounters an error (full, no connection, login error, etc.).
If 'check' no longer equals 'count' there was a failure, so it echoes error and exits the loop.
It subtracts 'count' from 'size' and copies that number of lines from the tail end of 'queue' over to a new file postfixed with '_new'.
It renames the _new file to the original queue file (I assume this overwrites it?).

I hope you dont mind me dissecting your coding,
I'm trying to learn from this as much as I can. ;)

I'm going to plop this into a test bed jail and run it, to see what comes out.
Thanks again, ill be back with results momentarily.

EDIT
I'm back, with some embarrassing results..
.. I cant get the script to run. :oops:
I'm supposed to save it as a .sh file and run it as such, right?
I just get a mess of errors when I try executing it.

1532723864660.png


The base OS in this jail is 'FreeBSD 11.1-STABLE' so that should run, right?
The shebang line says it is supposed to be a .sh script, I presume?

I'm terribly sorry for being this ignorant with freeBSD. :'‑(
 
Sokonomi said:
As it appears, freeBSD script coding looks eerily familiar with other types of coding ive tangled with,
Click to expand...
Note: this isn't specific "FreeBSD scripting" or coding, what you see here is a so called "Bourne shell script". At best you can call this "shell scripting", but even through we're using FreeBSD here that's actually not the important part. That's the shell itself (/bin/sh).

Sokonomi said:
So let me see if I interpreted it right;
It sets 'size' to 'queue' line count.
It runs a loop for every line in 'queue'.
It sets 'check' to 'count'.
It adds +1 to 'count' unless ftp encounters an error (full, no connection, login error, etc.).
If 'check' no longer equals 'count' there was a failure, so it echoes error and exits the loop.
It subtracts 'count' from 'size' and copies that number of lines from the tail end of 'queue' over to a new file postfixed with '_new'.
It renames the _new file to the original queue file (I assume this overwrites it?).
Click to expand...
That's about right, yeah. As you can see the diagram only covered the rough (overall) functionality, I usually don't bother myself with details when working out designs.

Sokonomi said:
I hope you dont mind me dissecting your coding,
I'm trying to learn from this as much as I can. ;)
Click to expand...
Oh right, silly me, I forgot to license this code! :p Naah, all good!

Sokonomi said:
I'm back, with some embarrassing results..
.. I cant get the script to run. :oops:
I'm supposed to save it as a .sh file and run it as such, right?
I just get a mess of errors when I try executing it.
Click to expand...
Well, file extensions don't matter on FreeBSD, what does matter are permission flags. The easiest way to get this to work is to place the script somewhere, make it executable using chmod +x test.sh and then try running it using: ./test.sh (this is assuming it's in the current directory).

However... sh ./test.sh should also work, with or without those flags. Making me question 2 things:
  • Is the script really still the same?
  • Is your shell (/bin/sh) actually the same Bourne shell?
Those error messages make little sense to me. So make sure there's no junk in the script, try: less test.sh to see what's actually in there and if that matches what I shared. Also: be sure to edit the variable values to match your own setup. I assume you already did so, but still wanted to mention it just in case.

You might also want to try: echo $ENV to see what it says, just in case.

Sokonomi said:
I'm terribly sorry for being this ignorant with freeBSD. :'‑(
Click to expand...
No need, we all got to start somewhere and I knew up front what I got myself into. Trust me: I wouldn't be doing this if I had the idea that I was wasting some of my time :)
 
Wow, you're so friendly, its nice to be met with hospitality for a change! :D
People have been pretty hostile for me asking 'stupid' questions on the freenas forums..

OH! I got it work now!
Very odd that it started working tough, all I did was exit the shell resave the script and try again.
Maybe it helped that I saved the script in notepad++ instead of regular notepad this time.
Now I'm actually getting feedback from ftp(1) in the terminal, so were getting somewhere.

However, It doesn't clear the last line off the queue file even though it did upload the file successfully,
so I think we might still be missing an EOF detection for exiting when the queue has been depleted entirely.

Maybe if we added
Bash: 
else
            echo "Queue depleted." > /dev/stdout;
            count=$((count+1))
            break;
fi
To the loop, would it fly?
Because if count equals size it means it went through all lines, right?

It also appears to be fussy about filenames that have spaces in them.
Ive tried encapsulating them in quotes but it doesn't seem to function that way.
So I guess the code is going to need some stuff to escape whitespaces?
Ive been banging on it all morning but I cant really figure out how to do this.
It seems to already be wrong when the line is captured in $a (its already missing part of the line there).

EDIT
After a whole lot of googling I can get it to escape the whole thing with sed but it still wont play nice with ftp.o_O
Bash: 
sed 's/ /\\ /g' files.txt > files2.txt
This seems to cough up the queue list with escaped whitespaces,
but the script wont properly use the whole line of this file either.
$a gets cut off too early for some reason; it always is just the first word, not the whole line.
I'm a little stumped on this one. :')

So lets say files.txt contains ..
Code: 
/video file 1.mkv
/video file 2.mkv
/video file 3.mkv
.. how do we make that work?
 
Sokonomi said:
OH! I got it work now!
Very odd that it started working tough, all I did was exit the shell resave the script and try again.
Maybe it helped that I saved the script in notepad++ instead of regular notepad this time.
Click to expand...
Most definitely. That's the part I forgot earlier; dos2unix(1) is a thing as well, there's a difference in the way Windows / DOS or Unix saves their files.

And that leads me to: how is your queue file generated? Because that might also be an important factor here.

Sokonomi said:
However, It doesn't clear the last line off the queue file even though it did upload the file successfully, so I think we might still be missing an EOF detection for exiting when the queue has been depleted entirely.
Click to expand...
Hm, that shouldn't happen but it is possible that the problem is related to the format of the queue file, I'll do some testing here as well.

You did spot one most definite bug which I overlooked (told you I had concentration issues right now due to the extreme weather, looking back I even added bugs in the UML diagram :D (which I left out of the script itself)):

Sokonomi said:
It also appears to be fussy about filenames that have spaces in them. Ive tried encapsulating them in quotes but it doesn't seem to function that way.
So I guess the code is going to need some stuff to escape whitespaces?

..CUT..

After a whole lot of googling I can get it to escape the whole thing with sed but it still wont play nice with ftp.o_O
Click to expand...
So to address my helpful mood just this once: there is a reason for that. One of it being the fact that we're messing with a favorite Unix aspect of mine: shell scripting. But the most important aspect is showcased right here. Instead of just reporting bugs and/or asking for help you're also actively participating and trying to come up with reasons why things go wrong and try to solve 'm on your own. That is a really good motivator for me to lend a helping hand. Or keyboard in this case I suppose :D

Anyway, the "spaces in filenames" is definitely an oversight on my end. It takes quotes, but more than one due to the loop we're running. In fact, I sometimes even rely on /bin/csh when it comes to processing such files because it has a bit better support for that. I'll avoid that for now but do me a favor and run: ls /bin/csh to see if that shell also exists on your system. If it does then I know that I can always resort to "plan B" if we need to (I doubt it).

Anyway, give me a few hours. Need to pick up some groceries first and then I'll address the "spaces in filenames" problem and see if I can find a cause for the queue file not to get processed correctly.
 
It might have been notepad saving under a different charset or some other oddball thing, but notepad++ seems to know shell. :p That little app is gods gift to coders.

As to where the queue is going to be coming from, I guess that should have been step 1. I'm hoping I can get the data out of sonarr via a custom script. If I can get another shell script (yay!) to append any new download to the tail end of the queue, I should be good to go. It shows here what variables you can extract from it, and it looks like an easy fish. Whenever the script triggers, it simply needs to slap sonarr_episodefile_path to the end of files.txt and shed be good to go. There's some snakes in the grass such as what happens when sonarr writes to it while the FTP shell script is actively processing the queue, but nothing too difficult to deal with I think.

As for the variables remaining up, this only seems to happen when I keep the shell open and run the script twice. They don't seem to flush once the script ends. Not sure if that would affect a cronjob, but its something to keep in mind.

And I love making things tick, automation is a passion for me, so writing shell scripts is basically a new toy. :-/ I do have a weird way of learning though. I tend to look at realworld examples to dissect and pick apart rather than digging around in textbooks and manuals first. Id rather disassemble an engine to see what makes it go, than to just look at diagrams all day. This kind of behavior apparently irks a lot of people, or they misunderstand how I go about things. I wouldn't ask people to do my dirty work, but I do love it when they show me something that I can retool and adapt, because I keep looking up the parts they used to do it.

csh is present in my bin, so we got plan B if we need it.

sed looks to be capable of escaping whitespaces, having it parse the queue to a new file seems to work nicely. So that's probably the thing I should be using?

I'm also having trouble figuring out where its getting $a from. Where/how does that variable get assigned? The lines need to get quoted, but since that's not happening, only the first string 'sticks' to $a. Ive tried manually slapping some quotes on the lines in the queue file itself, but that doesn't seem to work. I think cat might be blocking the quotes to make it safe.
 
Ok, I think ive almost cracked it, please verify;
Bash: 
#! /bin/sh

# Queue file (use full path!)
queue=files.txt
# FTP host
host=192.168.178.22:2121
# FTP username
user=sokonomi
# FTP password
pass=8520Derp
# FTP upload path
dir=/Removable/MicroSD

## Don't change stuff beyond this line
IFS=$'\n'
count=0
size=$((`wc -l $queue | cut -d ' ' -f8`+1))

for a in `cat $queue`; do
    check=0
       ftp -u ftp://$user:$pass@$host$dir/`basename $a` $a && check=1 && count=$((count+1))
    if [ ! $check ]
    then # Upload failed.
        echo "There was an error during the upload." > /dev/stderr
        break
    elif [ $count = $size ]
    then # End of queue reached.
        echo "Queue depleted." > /dev/stdout
        break
    else
        echo "Succeeded, moving on to next line." > /dev/stdout
    fi
done

newsize=$((size - count))
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue

Ive fixed the indexing problem, but Im hung up on how it registers an ftp failure.
It seems to set the check flag regardless of the ftp command returning 0.
The use of && is supposed to prevent that, right?
Any ideas? :p
 
Well, later than I anticipated (also had some IRL to sort out) but alas.

Plan B it is because /bin/sh isn't exactly the best shell when it comes to processing filenames with spaces in them, especially not when you're iterating over a file to grab the names. Unfortunately /bin/csh is mostly optimized for interactive use and not so much scripting. Resulting in a somewhat "messy" script.

It works, no problem, but produces some junk output and that will be noticeable when used with cron. Problem being that fuser sends output to both /dev/stdout and /dev/stderr. And unfortunately csh can only redirect stdout or stdout + stderr.

Still, this works.

Code: 
#!/bin/csh

# Queue file (use full path!)
set queue=files.txt
# FTP host
set host=breve
# FTP username
set user=anonymous
# FTP password
set pass=password
# FTP upload path
set dir=/upload

## Don't change stuff beyond this line

set count=0
set size=`wc -l $queue | cut -d ' ' -f8`

if ($size == 0) then
        echo "No files to upload."                              > /dev/stderr
        exit 1
endif


if (`fuser $queue | cut -d ':' -f2`) then
        echo "Queue file currently in use, exiting!"            > /dev/stderr
        exit 1
endif

foreach a ("`cat $queue`")
        set check=$count;
        set input=`echo $a | sed -e 's/\ /\\\ /g'`
        ftp -Vu ftp://${user}:${pass}@${host}${dir}/"`basename $input`" "$input"
&& @ count++
        if (${check} == ${count}) then
                echo "Something went wrong with the upload."    > /dev/stderr
                break
        endif
end

@ newsize = ${size} - ${count}
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue
So... this script checks that the queue file isn't being processed, iterates over all the filenames in the queue file and optionally escapes spaces in the name ("my song" vs. "my\ song"), then uploads the whole kaboodle.

The rest is pretty much unchanged.

Note: don't try to start this with sh ./<script>. If you insist on doing it like that then use csh, but best is to set the execution bit and then start the script directly.

About that $a: That's the work of for (last script) and foreach. Iteration, goes over the output of a command one line at a time (that's the idea at least) and then iterates; constantly replacing the value of a for the next line in the output.

Ever ran into a situation where rm * would tell you "too many arguments"? Probably not but that can happen in large directories. Before I knew about xargs(1) I usually solved that problem like this: for a in `ls`; do rm $a; done. Removes all the files in the current directory, only this time one by one.
 
That script doesn't seem to work?
It belts through the queue regardless of ftp failure, removing all but the last line from files.txt (that indexing issue again).
For some reason that AND operator isn't working?

From what I can gather, theres only 2 options if I want a decent feedback on FTP;
using grep to fish out clues of what is happening with ftp,
or simply using a better FTP package that actually gives usable returncodes.
 
I kept output minimal, you can remove the -V parameter behind ftp which will re-enable full logging again.

Unfortunately I cannot reproduce any weird issues on my end; I actually do test this stuff before sharing ;)

There is one caveat which I discovered after posting: if there's a ' in the filename then the script will complain because it picks that character up literally.
 
So if it fails to connect, it wont bilk through the entire queue without stopping?
(Cause it does that for me) :P
 
Then this happens on my end:
Code: 
$ ./upload
files.txt:
ftp: Can't lookup `account.no.way:ftp': hostname nor servname provided, or not known
ftp: Can't connect or login to host `account.no.way:ftp'
Something went wrong with the upload.
The script is very direct with this: the moment FTP exits with an error status of 1 or higher then count won't be increased in value and an error will be detected. There's really no way around that. I even tested it with using sh to start the script: sh ./upload, but that results in a simple hanging process which doesn't do anything.

So the best advice I can give you is to make sure the script file format is correct, csh is used or you use chmod +x <script> after which you start the script directly ( ./script).

Any problems beyond that I can only account to FreeNAS. That is the unknown factor here, at least for me.
 
I think I forgot to chmod the script, it appears to function now. Oops. O:‑)

Though I wonder if it would be better to see if theres a way for the script to adjust for changes to the queue file rather than just seeing if its in use. Because consider what happens if sonar shoves in a few new lines while the FTP script is the middle of uploading. The queue file will be longer than before the FTP script started, so it will end up cutting off more than just the ones it succeeded in uploading. Would it fix that issue if I added set size=`wc -l $queue | cut -d ' ' -f8` after the loop part so it refreshes the size for determining how much of a tail it needs to cut? Or is it possible to give head a negative number, so it ll always trim the correct amount from the top regardless?
 
:) Now that you've done it the hard way and learned a tonne about scripting, install Plex on the NAS and use the auto-sync feature of the Plex client on the tablet. ;)
 
phoenix said:
:) Now that you've done it the hard way and learned a tonne about scripting, install Plex on the NAS and use the auto-sync feature of the Plex client on the tablet. ;)
Click to expand...
That would require one of those ridiculously expensive plex pass accounts, I think? :-/Plex wont let me download the shows onto my tablet unless I do. Theres a way of having plex send new episodes down the tube automatically, serverside? I also don't see how it would be syncing 8Tb's worth of shows to a tablet that has just about 90Gb to play with. :p

I might also need to circle back to the script not stopping even when I use chmod on it.
1532976423968.png


It will still churn through the list when it gets a server timeout, and act as if all of them were successful, deleting them off the queue file (except the last one, theres still an indexing issue).
 
Sokonomi said:
I might also need to circle back to the script not stopping even when I use chmod on it.
Click to expand...
Aha, now that makes somewhat more sense. I assume you defined those ports by using them behind your host specification? Although I don't really get why you'd host an FTP server on another port than 21, it seems plausible that the script picks up the colon and port in a different way (though I doubt it). Anyway, this is a situation I can easily test with on my jail.
 
It works when the server is actually up. ;) I was just testing what it does with errors.
I haven't tried what it does when the server actually kicks back a "disk full" error, but ftp(1) somehow gives an exit status 0 when the connection fails.
Ive googled it a bunch of ways, and I keep coming up with people using grep to get FTP status.
But by all accounts, the exit code should not be 0 when it comes out of a failed connection, right?

EDIT:
Ive kept on digging and found that shell will still think ftp(1) ran successfully regardless of it connecting or not (it did its job after all).
I found an alternative solution called NcFTP which might help me get a better handle on this thing.

An example would be;
Bash: 
ncftpput -u $user -p $pass -P $port $host $dir $a
It returns a usable diagnostic code:
0 Success.
1 Could not connect to remote host.
3 Could not connect to remote host - timed out.
4 Transfer failed.
5 Transfer failed - timed out.
6 Directory change failed.
7 Directory change failed - timed out.
8 Malformed URL.
9 Usage error.
10 Error in login configuration file.
11 Library initialization failed.
12 Session initialization failed.

Basically anything but 0 still means it screwed up, but perhaps this is more predictable?
Code 4 might be used to delete last attempted file on the remote side as well,
or else it would leave incomplete files on the server, I think.
 
Why don't you just awk(1) the STATUS returned from the ftp(1) command/run for each file?
Seems like it's be real easy. Say; use a conditional while / if / unless
In Perl speak:
Code: 
unless ( $STATUS ) { do-something; } else { $VOMIT; }
where $STATUS was defined as the return code (number) for an ftp fail.
You could awk the status number returned from the ftp command(s), and place the failure number in $STATUS. But dropping any other ftp status number.

Just a thought. :)

--Chris
 
You must log in or register to reply here.
Back
Top