Dump-ing to a FTP server

[frabron]

Hi,

Our hoster has put some shiny new FTP servers to our use and I want to put my dump backups on them.Can I simply change the example for dump over ssh to put it on the ftp server? Like
# dump -C16 -b64 -0uanL -h0 -f - / | gzip -2 | ftp [url=ftp://user@backup.something.com/root.dump.gz]ftp://user@backup.something.com/root.dump.gz[/url]?
Probably it ain't so simple (it never is ) Or is using something like sysutils/fusefs-libs a better alternative to piping something over ftp?
And another question remains: How does FreeBSD handle incremental dumps? Does it need access to the initial dump to determine which incremental parts need to be backed up?

 

[worldi]

The script below takes input from STDIN and writes it to destfile on the FTP server. For example, zfs send whirl/data@backup | stream2ftp [email=user@example.com]user@example.com[/email] /backups/whirl-data.$(date +%F).zfs.

#! /bin/sh
usage="Usage: stream2ftp user@host destfile"
: ${1?$usage}
: ${2?$usage}

exec 3<&0
cat << FTPCMDS | ftp "$1"
  put "| cat <&3" "$2"
FTPCMDS

About incremental dumps: As long as -u is specified dump() keeps track of its actions in /etc/dumpdates.

 

[frabron]

Hey, thank you! I just tried your script, which I never could have made up myself, and it works! Although I'm not sure if it works as intended, since there're some messages that may hint to problems. And the duration of the upload is somewhat longuish. For testing purposes, I tried to upload /var/log/auth.log. My command:
# cat /var/log/auth.log | /root/bin/stream2ftp user@server /auth.log
The output:

# I'll skip the handshake bla ...
230-OK. Current restricted directory is /
230 38 Kbytes used (0%) - authorized: 104857600 Kb
Remote system type is UNIX.
Using binary mode to transfer files.
local: | cat <&3 remote: /auth.log
227 Entering Passive Mode
# it stalls here with no apparent actions happening
ftp: Can't connect to `The_ip:57406': Operation timed out
200 PORT command successful
150 Connecting to port 33213
226-77 Kbytes used (0%) - authorized: 104857600 Kb
226-File successfully transferred
226 0.002 seconds (measured here), 24.84 Mbytes per second
39740 bytes sent in 00:00 (23.77 MiB/s)
?Invalid command.
221-Goodbye. You uploaded 39 and downloaded 0 kbytes.
221 Logout.

What bothers me a little is ftp: Can't connect to `The_ip:57406': Operation timed out, ?Invalid command. and the duration of the upload of the small file - it's taking minutes instead of seconds. I guess it is related to the first highlighted error message with the 57406 port? Is this a problem on my side? pf is running, but stopping it doesn't help with the issue. Or shall I contact my hoster?

 

[wblock@]

Please see dump(8) about incremental backups. I think there is something about the typical multilevel backups in the Handbook, but can't find it right now. I personally only use full backups.

 

[worldi]

What bothers me a little is ftp: Can't connect to `The_ip:57406': Operation timed out, ?Invalid command. and the duration of the upload of the small file - it's taking minutes instead of seconds. I guess it is related to the first highlighted error message with the 57406 port? Is this a problem on my side? pf is running, but stopping it doesn't help with the issue. Or shall I contact my hoster?

The timeout occurs because either you or the other side (or something in between) is blocking the mentioned port. The invalid command error could be caused by the last line ("FTPCMDS") having leading spaces or tabs (hmm, cut'n'paste to vim without setting paste mode? ).

 

[kpa]

Passive mode FTP uses a secondary connection from a random TCP port on the client to a TCP port on the server's end. The TCP port on the server side is configured by the server and can be just about any port in the unprivileged 1024-65536 range. Check your firewall settings and make sure the secondary connection gets trough to the server.

 

[frabron]

Thanks for all your answers. I've contacted my hoster concerning the passive mode ports. I'm not blocking anything (consciously) on my machine, turning off the firewall ( pf) doesn't change the situation so I'm aiming at their direction.

The invalid command error could be caused by the last line ("FTPCMDS") having leading spaces or tabs (hmm, cut'n'paste to vim without setting paste mode?

good guess, thanks for spotting that without seeing my script