Archiving and Compression Tools

Hello all,

I wasn't really exactly sure where to post this, so if it's the wrong forum, please let me know.

Anyway, on to what I'd like to find out:

I'd like to know more about the Archiving and Compression Tools available on FreeBSD.

Information about my set up :

My Wife and I have a pretty large number of Computers in our Home, and we run a multitude of OSs here, which includes this:

FreeBSD (Mostly 9.0-RELEASE, and for this question, it's all that matters)

Linux - Slackware, Debian, OpenSUSE, Mandriva, LinuxConsole, NetSec L, a few others here and there, but they don't matter.

Windows - I have a single Windows Partition on ONE Computer, which is mainly for games, since, really, what other use would I have? ;) My Wife also has a Windows Partition. Both are Windows 7 Home Premium 64 Bit Edition.

Solaris.

Anyway, I'm interested in learning more about the tools available for Archiving and Compression, and, of course, anyone who would like to share their thoughts and experience and opinions, is very much welcome to do so.

I know there are a LOT of tools out there for this, and I've been trying to test out as many as I can, but I'd like some help if possible.

Now, the most important thing to ME, is the Compression Rates; I have a Server set up running Slackware Linux 12.0, and it's served me well over the last couple years. For the most part, and for the sake of Cross Platform ease of use, I've been using Bzip2 on that.

On Windows 7, I've installed TUGZip, PeaZip, and 7-Zip, which I actually like a lot. I also have WinRAR installed but I don't really use it, as I prefer the way the others work. But again, I'm going to use whatever works best after I learn more.

For Archiving, I've been keeping it oldschool and just using Tar. So with that particular aspect, I'm open to ANY Suggestions.

What I'd like to know, is the following if possible:

what is the absolute BEST Compression? I've had a lot of people tell me different things, and when I use my Server to take the load of Compressing multiple huge files, I generally just do this; bzip2 -v9 File;bzip2 -v9 OtherFile;bzip2 -v9 AnotherFile And so on.

I know I'm probably NOT getting the best possible space saving doing this, and so one thing I'd like to hear more from all of you about, is what I could do to get better Compression.

One of the main reasons I'm trying to do this, is that we have a limited budget, and even with Disk Space being so cheap, I still can't just go buy myself some huge multi Terabyte drive array, though I'd LOVE to.

The next thing I guess, is what Compression tools do all of you use? What have you had the best luck with?

My Server is Slackware 12.0, and it's very stripped down, and I don't have X installed, or anything like that, as I don't need that on a machine that rarely has a monitor turned on anyway, and I also don't understand why someone would actually want a GUI on a Server anyway, as that's RAM and CPU wasted on something that isn't required.

The types of files I'm compressing, are of all types. The reason I set this Server up was because I wanted a central machine I could upload Back Ups to. Basically, I took the very first Computer I ever bought, and because it was my main desktop for years, but the video card was starting to have issues in it, I thought "Why not just set it up as a Server since the GUI is the only area you notice the Video Card is having issues?" And so I did that.

The HD it came with, is a little odd because it's 43 GB. I took that HD out, and installed a brand new 120 GB HD, and then, installed the old HD it came with, and then did this:

I made the new HD the main Master, and put my Swap Partition on it, and then of course, my Root File System and all that. The old HD, the one it came with, which is 43 GBs, I decided to just simply make it one big Partition, and mounted it as "/storage" and then I made it so that all logged in users could read and write to and from it. This totally solved my problems with back ups.

Having so many machines made back ups a nightmare, and so, I decided to set it up with VSFTPd, and SSHd, and then, I made a user account to be used for logging in and uploading my back ups to. This actually has worked very well for me, and I wanted to make sure to have a valid user account, as I didn't want anonymous, and so anyway, before I go to far off topic, the facts are that I'm starting to run really low on disk space.

I have my Music Collection, which, a lot of them are MP3s, and then I also have Ogg Vorbis as well, and then a bunch of videos ranging from WMVs, AVIs, and MP4s, and then pictures as well in a variety of formats, and A LOT of PDFs, text, and so on. Basically, all the Data I think is Important, from every machine we have heh.

So basically, I'm just wondering if anyone would like to share some information or experience or opinions on the way they Archive and Compress their Data, and the Tools they use to do it. I do like 7zip and stuff, but I'm just wondering what all I should try.

The only thing that's really important, is the best Compression possible. If it happens to work on other things and is Cross Platform, that'd be a plus :)

Thanks all,

-gore
 
How about reinstalling that machine with FreeBSD and use ZFS as filesystem for "/storage" and activate gzip-9 compression on that filesystem? That way, everything you put in there is automagically compressed without you having to do anything besides putting the data in. Beware of how hard gzip-9 presses the cpu so copying data in there(over ftp, NFS, SAMBA, etc) will be far from fast, but you won´t have to manually handle the compressing, it just happens. And paranoid as I am, I would also suggest buying/scrambling up another 120GB hard drive to make "/storage" mirrored, since we are talking about backups.

/Sebulon
 
Sebulon said:
How about reinstalling that machine with FreeBSD and use ZFS as filesystem for "/storage" and activate gzip-9 compression on that filesystem? That way, everything you put in there is automagically compressed without you having to do anything besides putting the data in. Beware of how hard gzip-9 presses the cpu so copying data in there(over ftp, NFS, SAMBA, etc) will be far from fast, but you won´t have to manually handle the compressing, it just happens. And paranoid as I am, I would also suggest buying/scrambling up another 120GB hard drive to make "/storage" mirrored, since we are talking about backups.

/Sebulon

Whoa, dude.... I like that idea! I still want to wait and see what else the members here reply with, but I may set up a test machine with your idea, because I really REALLY like that. The Network being slowed down isn't an issue for me, I don't mined that at all as long as it works.

See, THIS is one of the many reasons I came here to ask, and not another forum; You guys are seriously Brilliant. I've never personally used ZFS, but I'm into OSs in general, and I'm also interested in File Systems as well, and always have been.

I've always found different File Systems interesting. I think it's the way they work differently and yet achieve basically the same thing that grabs my attention. But anyway just so I don't start to ramble on (Which I'm known to do) I just wanted to say thank you very much for the great idea, and, I look forward to more replies :)

by the way; Other than gzip, do you personally use any other tools? I'm very interested in any ideas and information anyone is willing to share with me :)
 
gore said:
Whoa, dude.... I like that idea!

by the way; Other than gzip, do you personally use any other tools? I'm very interested in any ideas and information anyone is willing to share with me :)

No problem, glad you like it:)

Personally, I just use the default compression in ZFS, which is called lzjb. It provides a decent ratio, depending on what the data is of course, while not being as overly "hard" as gzip is. We use ZFS for general user storage over SAMBA and AFP, and VM storage over NFS. At one of our sites, they store 1TB of VM data in only ~500GB using lzjb and write speed is 200MB/s. If you change lzjb to gzip-9, throughput drops to 5-10MB/s; that´s how hard it is. And I think that is sacrificing more in performance than you get back in savings.

In my NAS at home, where I store lots(3.5TB) of uncompressible files (images, movies, music) however, storage savings are nill with lzjb, so if you want those kinds of files to have any saving, you may actually need a "harder" algorithm. But, is it worth it...

/Sebulon
 
With regard to the central server, I would have installed the OS on the old disk to save space on the primary disk for /storage. However I would consider installing on such machine a system that is built to be a NAS, since it seems to me that is the only purpose of such machine. A FreeNAS is a good example of distribution, and gives you ZFS with compression and can boot even out of an USB key (that is you can save even more disk space for real storing data).

About compression I use bzip2, and I'm fine with that.
For backing up I would suggest to investigate an automated solution like bacula, or at least do a centralized set of scripts that will connect to any of your computers and do an incremental backup maybe using rsync. I did it in a FreeNAS installation, so that every night the server connects thru ssh to a machine and does an rsync of some files.
 
Sebulon; Yes I did. I've been kinda sick the last couple days to the point I literally couldn't even sit in my Computer Chair which was purchased because of my back in the first place, but it's been REALLY bad. So I'm sorry for taking so long to show the gratitude but yea been a rough week.

Fluca; Yea I probably should have explained why I am using a PC for that; Basically I just lost my job, so I can't exactly go and buy one, though we HAVE made our upgrade plans and things for when we get out of this crap, and other than me wanting an old SGI workstation to play with (I've always wanted to toy with IRIX, and I'm into OSs a lot, so I've wanted to at least toy with it for a long time) and the other main thing on there is a Sparc, and a NAS set up like you're describing.

The Sparc is for my Wife mainly because She Loves Sun as I do SGI heh. Besides, they're pretty cheap for a "toy" one to just play with and stuff. I also found an old Sun Processor I wanted to make into a Necklace but I couldn't order it.

Anyway sorry it took so long to respond but thanks again both of you.
 
Back
Top