Backup large files to tape

G

grantd

Hey Everyone,
I recently moved my backup server to FreeBSD from Windows Server2008 R2 because I was having a nightmare of a time backing up to tape. I read a bunch on info online on how to write backups using pas, tar, cpio, and afio, but none of them seem to work with large files (at least in my experience). I have even written a Perl script to write each file to the tape and record its track offset and everything to a log, and allow for incremental backups based off a custom written database file. My only problem is writing large files to tape.

I have files that are 4.5 GB (home made movie DVDs) and some backup data files that can exceed 60 GB and I want to be able to write this data to LTO4 tapes. Now, I do not need the backup format to allow spanning of tapes since I can have my script do that for me, but I do need it to be able to write the data to the tape. The only success (it seems like success, not sure if it really worked) has been using cat to output to the tape, but not sure how to read the data back to a file (every try has resulted in odd sized files of only a few MB).

Any suggestion? I have read that in Linux, there is a tar-E option to enable such support, but haven't found it in FreeBSD yet.

Thanks in advance,
Grant
 
Savagedlight said:
Have you tried using dump yet?
Click to expand...

I want to be able to restore individual files, and dump appears to only allow full file system restores. If I am wrong, can you point me to a good resource on how to do backups and restores of individual files with dump/restore?




wblock said:
Tape devices shouldn't have a file size limit, but bsdtar(1) very well might. The man page doesn't say anything about it, though. Best to ask on the mailing lists, like freebsd-questions.
Click to expand...

I can tar it to a file just fine, I only have a problem going to a tape. I can also tar it to the console and pipe it to the tape with cat, but then restoring the file results in a truncated archive error.
 
Hi there,

I'm not familiar with BSD tar, but I'll try to help using GNU tar as a reference (the one on linux). Your issue could be with your tape block size and tar block size. tar by default writes in blocks of size 10 kb. This usually does not match with the block size setting of your tape drive.

You can use the mt command to get your current tape drive setting. For example
mt -f <device> status

If you think that could be your problem. Do one of the following
mt -f <device> blocksize 0 (This would make your blocksize setting as zero)
or
mt -f <device> blocksize 10k (assuming that your tar command is writing in 10k blocks)
or
tar -b <blocksize> (size in 512 bytes records. i.e for 1k -b 2)

If these do not help, try posting the output when you try tar.
 
To explain the mt commands above, the first option sets the tape drive to variable block size mode. So any block size will be fine. The second tries to match the tape drive blocks size with the block size tar is writing with. The third tries to match tar's block size with the tape drives block size.
 
If you can [cmd=""]cat[/cmd] to tape, and [cmd=""]tar[/cmd] to file, then why not just [cmd=""]tar[/cmd] directly to tape?

[cmd=""] tar cvf /dev/TAPE[/cmd] or something similar.

Or am I missing something here?
 
Filesystem level:

You can use an interactive restore(8) "restore -i" to select particular files/directories from a dump file.

You can also restore a filesystem from a dumpfile onto a file or memory backed disk using mdconfig (if resources permit); then mount it and navigate around.

File level:

For file-level archiving/backup try using pax(1). It allows for differing formats (cpio/ustar/bcpio etc) with the "-x" switch
 
Thanks for the suggestions. I haven't had a chance to try anything since my tape drive broke. I am waiting to hear back from Quantum on the replacement drive. Once I get it back, I will test out a few things.

As for Bacula, I couldn't figure out how to set it up for a single drive and single server (seemed over complex for a simple setup).
 
Sorry for the long delay in reply, I just got the replacement drive yesterday.

So I set the block size to 10k with the following command:
mt blocksize 10240

And defiles the blocksize in TAR:
tar -b 20 -cvf /dev/nsa0 [file]

and still get the same error:
Code: 
tar: Removing leading / from member names
a [file]tar: Write error

This is all I am going off of, so I guess you all can understand my confusion.

As for dump, I want the ability to select what files I am backing up as well. The full backup will be something around 5-6 TB running off of a ZFS RAID-Z partition comprised of eight 1.5 TB drives.
 
For LTO drives normally you want a large block size, at least 64K but for performance more optimatally 256K or larger. Give those two block sizes a try, and if it's still giving you problems you could also try archivers/gtar from the ports tree...

thanks Andy.

PS regarding your hardware failure, be aware LTO drives aren't very resilient to under-streaming of data, sometimes called shoe shinning. That is, you are meant to supply sufficient data to the drive to keep it writing without it having to constantly stop the tape, rewind, reposition and wait for more data. Shoe shining results in an early death for LTO drives...
 
I tried 64K, 128Km 256K, 512K, and 1024Km each with similar or the same errors.

On GTAR, it reports that the device it out of space when it is a freshly erased tape. With sizes of 64K or higher, it has an unrecoverable error.
 
So I have tried all these suggestions with no luck. Last night, I boot to the live Arch Linux DVD and installed mt-st in the live environment. After a few tests, I successfully used tar to backup, list, and restore data from the tape and used mt to move between records, erase, and rewind. Not sure why this doesn't work in FreeBSD, but figured I would post here to give everyone an update.

Thanks for the support, but I am migrating my backup server to Arch Linux after all this trouble. I may try future BSD releases when they come out.
 
You must log in or register to reply here.
Back
Top