Problem with transferring huge file

[nfm]

I've problems transferring a database file which are around 80 gb (one file) from a Linux box to a FreeBSD box. I've tried with most of the common tools such as rsync, ftp, nfs and scp. None of these managed to transfer the whole file. The record was just over 50 GB.

It seems that this accur only when a FreeBSD box are involved, between two Linux servers works fine.

Both boxes are HP Proliant DL380 G5 servers with dual quad-core Xeon and 24 GB of ram.

FreeBSD box are running FreeBSD 7.0-RELEASE with ufs-filsystem

Linux box are running Redhat Enterprise Linux 4 with ext3-filesystem.

I've tried with some sysctl tweaking but this are very unexplored areas for me so my values can be completely wrong.

net.inet.tcp.recvspace=393216
net.inet.tcp.sendspace=393216
net.inet.tcp.delayed_ack=0
net.inet.udp.recvspace=147456
kern.ipc.maxsockbuf=1048576
kern.ipc.nmbclusters=49152

[ale]

I think that you could try with split(1)
split -b 10240m $HUGE_DB_FILE db.part.

This should split your file in a number of 10GB files named db.part.$SOMETHING
You can tune the size using the -b argument, specifying k/m suffix as kb/mb.
After you have transferred all the files on the destination box
cat db.part.* > $HUGE_DB_FILE

I know, this doesn't explain why the transfer isn't working, but it could be a workaround to have the work done.

[Lowell]

When the transfer fails, what are the symptoms?

[I don't have enough free space around to experiment with reproducing the problem.]

[nfm]

The transfer just simply drop to 0kb/s and never get up to speed again. I waited for at least an hour.

I checked the dmesg log after the transfer stalled but no new messages there.

[ale]

Is there something between the two boxes?
Can you try with a cross cable?

[nfm]

The only thing between are a HP gbit switch (don't know model).

Try a cross over cable are sadly not possible, the servers are located in a datacenter where I have to book an airline ticket too get to.

The split have been a good idea but I just realized that I haven't that much space left on the server.

Thanks for your help anyway. :-)

[Andrius]

What about resuming ftp (or whatever else) transfer? Also bittorrent is quite good to transfer huge files and even if it failed in the middle you can just restart the client and continue from where it left.

Well, that is if you can use bittorrent clients on both machines.

[danger@]

so the problem was actually space related? Good to know.

[nfm]

No, the problem are not space related. I Only have to little space for the solution where a split was suggested in the frist reply.

[ale]

I was suspecting something like that, I mean about the space needed to copy all the splitted files and then rejoin them.
So you could try a script that do more or less something like that, from the directory where the splitted files are:
#!/bin/sh
for F in $( ls -1 db.part.*);
do
cat "$F" | ssh $USER@$REMOTEHOST "cat >> $FULL_PATH_TO_THE_HUGE_FILE"
done
so the files get "transferred" one by one and appended to the destination.

Obviously you have to configure your ssh so you can authenticate using keys or you'll be prompted for the password for each file.

[joel@]

Split may be a workaround but there seems to be a real problem here.

nfm: if you don't get a good answer or a solution within a few days, try emailing directly to stable@FreeBSD.org and describe your problem with as much details as you can.

[danger@]

and also testing it with 7.1 would be good too...

[aragon]

It would be useful to get more information on what happens when you attempt to perform the transfer. Does any part of the data transfer at all? What happens over the wire? (tcpdump)

[Mel_Flynn]

Somewhere in the -questions archive, I remember a post by someone that kept this patch (http://www.andrew.cmu.edu/user/mstevens/hpn-ssh/) maintained across versions.
I know, it's vague, but I can't find it at the moment :/

[dh]

It's probably the NIC that's acting up, check dmesg or /var/log/messages for NIC related stuff. Try changing NIC media type - when media is 1000baseTX, try changing to 100baseTX (sure it's slower, but it might help).
You might also try running tcpdump when the file transfer speed reached 0kB/s and see if any packets are moving at all (or sending out packets but not receving ACK-s back).
BTW, does the file transfer becomes stalled always at the same place or is that completely random?

[liamjfoy]

Can you write back the results of:

ifconfig -a
netstat -s ip
netstat -s tcp

what do logs for openssh show on both sides? BTW, if you can show me these results after rebooting both machines and just starting the transfer?

[nfm]

I'm sorry for not giving you an update.

I finally did the "split"-solution to get the job done because the old server was running out of diskspace which was the main reason for the move.

I'll see if my company has a pair of spare servers I can use for more testing to get this solved.

[r-c-e]

Is FTP failing even with resume enabled? If so, perhaps its a file system issue rather than a network issue.

[cracauer@]

This reminds me of what happens when a NIC driver screws up. I see that on wireless on a regular basis.