HOWTO: Modern FreeBSD Install RELOADED (vermaden way)

vermaden · #1 March 8th, 2010, 14:47

All these years sysinstall(8) was helping us to install FreeBSD with most options we needed, today with new filesystems/features like GJournal/ZFS/Geli/GMirror/GStripe its no longer up to the task, because it only supports creating installation on UFS filesystem with SoftUpdates turned ON or OFF.

In this guide you will learn how to setup FreeBSD installation in simple yet flexible setup based on read-only UFS (without SoftUpdates) for 'base system' [1], some SWAP space, /tmp mounted on SWAP and all the other filesystems (/var /usr ...) mounted on ZFS. It will not require rebuilding anything, just simple setup on plain MBR partitions. I should also mention that we would be using AHCI mode for disks. I also provided two versions, for system with one harddisk and with three of them for redundant setup.

Here is the layout of the system with 1 harddisk:

Code:

MBR SLICE 1 |    / | 512 MB | UFS/read-only 
            | SWAP |   2 GB |
            | /tmp | 512 MB | mounted on SWAP with mdmfs(8)
------------+------+---------------------------------------
MBR SLICE 2 | /usr |   REST | ZFS dataset
            | /var |   REST | ZFS dataset

... and here layout for single disk for system with 3 disks:

Code:

MBR SLICE 1 |    / | 512 MB | UFS/read-only 
------------+------+--------+------------------------------
MBR SLICE 2 | SWAP |   1 GB |
            | /tmp | 512 MB | mounted on SWAP with mdmfs(8)
------------+------+--------+------------------------------
MBR SLICE 3 | /usr |   REST | ZFS dataset
            | /var |   REST | ZFS dataset

Redundancy planning for system with 3 disks:

Code:

 [ DISK0 ]           [ DISK1 ]           [ DISK2 ]
 [   /   ] < RAID1 > [   /   ] < RAID1 > [   /   ]
 [ SWAP0 ]           [ SWAP1 ]           [ SWAP2 ]
 [   Z   ] < RAID5 > [   F   ] < RAID5 > [   S   ]

FreeBSD core, the 'base system' [1] should remain almost unchanged/untouched on daily basis while you can mess all other filesystems, this ensures that when things go wrong, you will be able to fix anything still having working 'base system' [1].

You will need *-dvd-* disk or *-memstick-* image for this installation, *-disk1-* will not do since it does not contain livefs system.

Here is the procedude, described as simple as possible.

1.0. I assume that our disk for the installation would be /dev/ad0 (/dev/ad0 /dev/ad1 /dev/ad2 for system with 3 disks)

1.1. Boot *-dvd-* from DVD disk or *-memstick-* image from pendrive

Code:

Country Selection --> United States
Fixit --> CDROM/DVD (*-dvd-*) or USB (*-memstick-*)

1.2. Create your temporary working environment

Code:

fixit# /mnt2/bin/csh
# setenv PATH /mnt2/rescue:/mnt2/usr/bin:/mnt2/sbin
# set filec
# set autolist
# set nobeep

1.3. Load needed modules

Code:

fixit# kldload /mnt2/boot/kernel/geom_mbr.ko
fixit# kldload /mnt2/boot/kernel/opensolaris.ko
fixit# kldload /mnt2/boot/kernel/zfs.ko

1.4. Create/mount needed filesystems

Code:

DISKS: 3                                               | DISKS: 1
# cat > part << __EOF__                                | # cat > part << __EOF__
p 1 165 63  512M                                       | p 1 165 63  2560M
p 2 165  * 1024M                                       | p 2 159  *     *
p 3 159  *     *                                       | p 3   0  0     0
p 4   0  0     0                                       | p 4   0  0     0
a 1                                                    | a 1
__EOF__                                                | __EOF__
                                                       |
# fdisk -f part ad0                                    | # fdisk -f part ad0
# fdisk -f part ad1                                    |
# fdisk -f part ad2                                    |
                                                       |
# kldload /mnt2/boot/kernel/geom_mirror.ko             |
# gmirror label  rootfs ad0s1                          |
# gmirror insert rootfs ad1s1                          |
# gmirror insert rootfs ad2s1                          |
                                                       |
# bsdlabel -B -w /dev/mirror/rootfs                    | # cat > label << __EOF__
                                                       | # /dev/ad0s1:
                                                       | 8 partitions:
                                                       |   a: 512m  0 4.2BSD
                                                       |   b: *     * swap
                                                       | __EOF__
                                                       |
                                                       | # bsdlabel -B -w ad0s1
                                                       | # bsdlabel       ad0s1 | tail -1 >> label
                                                       | # bsdlabel -R    ad0s1 label
                                                       |
# glabel label swap0 ad0s2                             | # glabel label rootfs ad0s1a
# glabel label swap1 ad1s2                             | # glabel label swap   ad0s1b
# glabel label swap2 ad2s2                             |
                                                       |
# newfs /dev/mirror/rootfsa                            | # newfs /dev/label/rootfs
# zpool create basefs raidz ad0s3 ad1s3 ad2s3          | # zpool create basefs ad0s2
# zfs create basefs/usr                                | # zfs create basefs/usr
# zfs create basefs/var                                | # zfs create basefs/var
# mkdir /NEWROOT                                       | # mkdir /NEWROOT
# mount /dev/mirror/rootfsa /NEWROOT                   | # mount /dev/label/rootfs /NEWROOT
# zfs set mountpoint=/NEWROOT/usr basefs/usr           | # zfs set mountpoint=/NEWROOT/usr basefs/usr
# zfs set mountpoint=/NEWROOT/var basefs/var           | # zfs set mountpoint=/NEWROOT/var basefs/var

1.5. Actually install needed FreeBSD sets

Code:

# setenv DESTDIR /NEWROOT
# cd /dist/8.0-RELEASE

# cd base
# ./install.sh (answer 'y' here)
# cd ..

# cd manpages
# ./install.sh
# cd ..

# cd kernels
# ./install.sh generic
# cd ..

# cd /NEWROOT/boot
# rm -r kernel
# mv GENERIC kernel

vermaden · #2 March 8th, 2010, 14:47

1.6. Provide basic configuration needed to boot new system
1.6.1.

Code:

DISKS: 3                                          | DISKS: 1
# cat > /NEWROOT/etc/fstab << __EOF__             | # cat > /NEWROOT/etc/fstab << __EOF__
#dev                #mount #fs  #opts #dump #pass | #dev              #mount #fs  #opts #dump #pass
/dev/mirror/rootfsa /      ufs  rw    1     1     | /dev/label/rootfs /      ufs  rw    1     1
/dev/label/swap0    none   swap sw    0     0     | /dev/label/swap   none   swap sw    0     0
/dev/label/swap1    none   swap sw    0     0     | __EOF__
/dev/label/swap2    none   swap sw    0     0     |
__EOF__                                           |
                                                  |
# cat > /NEWROOT/boot/loader.conf << __EOF__      | # cat > /NEWROOT/boot/loader.conf << __EOF__
zfs_load="YES"                                    | zfs_load="YES"
ahci_load="YES"                                   | ahci_load="YES"
geom_mirror_load="YES"                            | __EOF__
__EOF__                                           |

1.6.1.

Code:

# cat > /NEWROOT/etc/rc.conf << __EOF__
zfs_enable="YES"
__EOF__

1.7. Unmount filesystems and reboot

Code:

# cd /
# zfs umount -a
# umount /NEWROOT
# zfs set mountpoint=/usr basefs/usr
# zfs set mountpoint=/var basefs/var
# zpool export basefs
# reboot

Now lets talk things you will need to do after reboot.

2.0. At boot loader select boot into single user mode

4. Boot FreeBSD in single user mode

Code:

Enter full pathname of shell or RETURN for /bin/sh: /bin/csh
% /rescue/mount -w /
% /rescue/zpool import -D || /rescue/zpool import -f basefs
% exit

2.1. Login as root without password

Code:

login: root
password: (just hit ENTER)

2.2. Set root password

Code:
# passwd

2.3. Set hostname

Code:

# echo hostname=\"HOSTNAME\" >> /etc/rc.conf

2.4. Set timezone and date/time

Code:

# tzsetup
# date 201001142240

2.5. Tune the ZFS filesystem (only for i386)

Code:

# cat > /boot/loader.conf << __EOF__
vfs.zfs.prefetch_disable=0      # enable prefetch
vfs.zfs.arc_max=134217728       # 128 MB
vm.kmem_size=536870912          # 512 MB
vm.kmem_size_max=536870912      # 512 MB
vfs.zfs.vdev.cache.size=8388608 #   8 MB
__EOF__

2.6. Mount /tmp on SWAP

Code:

# cat >> /etc/rc.conf << __EOF__
tmpmfs="YES"
tmpsize="512m"
tmpmfs_flags="-m 0 -o async,noatime -S -p 1777"
__EOF__

2.7. Move termcap into /etc (instead of useless link on crash)

Code:

# rm /etc/termcap
# mv /usr/share/misc/termcap /etc
# ln -s /etc/termcap /usr/share/misc/termcap

2.8. Add latest security patches

Code:

# freebsd-update fetch
# freebsd-update install

2.9. Make all changes to configuration in /etc, then set / to be mounted read-only in /etc/fstab

Code:

DISKS: 3                                           | DISKS: 1
 #dev                #mount #fs  #opts #dump #pass |  #dev              #mount #fs  #opts #dump #pass
+/dev/mirror/rootfsa /      ufs  ro    1     1     | +/dev/label/rootfs /      ufs  ro    1     1
-/dev/mirror/rootfsa /      ufs  rw    1     1     | -/dev/label/rootfs /      ufs  rw    1     1
 /dev/label/swap0    none   swap sw    0     0     |  /dev/label/swap   none   swap sw    0     0
 /dev/label/swap1    none   swap sw    0     0     |
 /dev/label/swap2    none   swap sw    0     0     |

2.10. Reboot and enjoy modern install of FreeBSD system

Code:

# shutdown -r now

vermaden · #3 March 8th, 2010, 14:48

To summarise, this setup provides us these things:

-- bulletproof 'base system' [1] on UFS (w/o SU) mounted read-only
-- /tmp filesystem mounted on SWAP
-- usage of new AHCI mode in FreeBSD
-- flexibility for all other filesystems on ZFS
-- fully working environment on crash (/etc/termcap)
-- disks/filesystems mounted by label, possible device name changes are harmless
-- RAID1 for / and RAID5 for all other systems on setup with 3 disks

[1] base system is / and /usr while this setup, in context of this setup I name 'base system'
the most important core of FreeBSD, the / filesystem and its binaries/libraries/configuration
(thanks to phoenix for reminding me what REAL base system is/means)

CHANGELOG

1.0 / 2010-01-14 / initial version
1.1 / 2010-01-15 / simplified PATH

+fixit# setenv PATH /mnt2/rescue:/mnt2/usr/bin -fixit# setenv PATH /mnt2/bin:/mnt2/sbin:/mnt2/usr/bin:/mnt2/usr/sbin

1.2 / 2010-01-15 / added link for termcap (instead of duplicate on /etc and /usr) [2.6.]

.# rm /etc/termcap +# mv /usr/share/misc/termcap /etc +# ln -s /etc/termcap /usr/share/misc/termcap -# cp /usr/share/misc/termcap /etc

1.3 / 2010-01-21 / removed unneeded mount commands [2.0.]

-# zfs mount basefs/var -# zfs mount basefs/usr

1.4 / 2010-03-08 / added setup for 3 disks + cleanup

too much to fit here, we can as well call this new version RELOADED

MIRROR THREAD: http://daemonforums.org/showthread.php?t=4200
POLISH VERSION: http://bsdguru.org/dyskusja/viewtopic.php?t=19392

ADDED: 2010/10/21

After rethinking setup from my HOWTO and after phoenix thoughts I currently use that setup for most FreeBSD installations that include ZFS.

LOGICAL SETUP

Code:

UFS 512m /           ro
ZFS *    /home       rw | atime=off
RAM 128m /tmp        rw | async
UFS *    /usr        ro | softupdates (mounted r/w only for packages updates)
ZFS *    /usr/obj    rw | atime=off | checksum=off
ZFS *    /usr/ports  rw | atime=off
ZFS *    /usr/src    rw | atime=off
ZFS *    /var        rw
UFS 128m /var/db/pkg ro | softupdates (mounted r/w only for packages updates)

PHYSICAL SETUP (LAPTOP w/ 1 DISK)

Code:

p1 8g disk0s1a 512m UFS /           newfs -m 1    /dev/label/root
      disk0s1e 128m UFS /var/db/pkg newfs -m 1 -U /dev/label/pkg
      disk0s1f    * UFS /usr        newfs -m 1 -U /dev/label/usr

p2 *g disk0s2  ZFS/home             zfs create -o mountpoint=/home      pool/home
               ZFS/var              zfs create -o mountpoint=/var       pool/var
               ZFS/usr              zfs create -o mountpoint=none       pool/usr
               ZFS/usr/src          zfs create -o mountpoint=/usr/src   pool/usr/src
               ZFS/usr/obj          zfs create -o mountpoint=/usr/obj   pool/usr/obj
               ZFS/usr/ports        zfs create -o mountpoint=/usr/ports pool/usr/ports

               (if You need SWAP, omit on CF/Pendrive/SSD disks)
               ZFS/SWAP             zfs create -V 2g                    pool/swap

RAM/SWAP 128m  /tmp                 tmpmfs=YES --> /etc/rc.conf

PHYSICAL SETUP (CF + DISKS)

Code:

8g CF    disk0s1a 512m UFS /           newfs -m 1    /dev/label/root
         disk0s1e 128m UFS /var/db/pkg newfs -m 1 -U /dev/label/pkg
         disk0s1f    * UFS /usr        newfs -m 1 -U /dev/label/usr

*g ZFS   ZFS/home                      zfs create -o mountpoint=/home      pool/home
         ZFS/var                       zfs create -o mountpoint=/var       pool/var
         ZFS/usr                       zfs create -o mountpoint=none       pool/usr
         ZFS/usr/src                   zfs create -o mountpoint=/usr/src   pool/usr/src
         ZFS/usr/obj                   zfs create -o mountpoint=/usr/obj   pool/usr/obj
         ZFS/usr/ports                 zfs create -o mountpoint=/usr/ports pool/usr/ports

         (if You need SWAP)
         ZFS/SWAP                      zfs create -V 2g                    pool/swap

128M RAM /tmp                          tmpmfs=YES --> /etc/rc.conf

Of course for serious storage/backup servers it would be 'nice' to have that CF (or pendrive) mirrored via GEOM/mirror.

graudeejs · #4 March 8th, 2010, 15:11

hmm, are you able to boot into single user mode with ZFS?
I can't for some reason, maybe because my HDD's are encrypted. But there are devices with eli (decrypted), weird

vermaden · #5 March 8th, 2010, 15:34

@killasmurf86

/ is on UFS (with bsdlabel) so no problem to boot into single user mode, I havent played with encrypted / to check here, maybe I will in some free time @ virtualbox.

graudeejs · #6 March 8th, 2010, 15:42

Encrypted root with UFS will work

, bin there done that

dewarrn1 · #7 April 4th, 2010, 06:56

First off, thanks to Vermaden for posting this! It's a very slick way to take advantage of the best filesystems that FreeBSD offers. I'm considering migrating a home server to 8.0, and this seems like a great setup for me. I've got a couple of questions before I give it a try, though.

First, would including a fourth disk be as simple as it looks? I've got 7.2's ZFS spanning 3 disks at the moment, but I've got a fourth sitting around and figure that it might as well be in the server.

Second, do you have any recommendations for maintaining data integrity during the move? I've got external HDD's that can hold all of my stuff, but they're just FAT32 and lack the kind of checksum protection that ZFS is giving me on the current system. My plan would be to use all three of the current disks and a fourth in the new system, but that will require wiping out the current filesystem.

Thanks in advance!

vermaden · #8 April 4th, 2010, 08:54

@dewarrn1

Quote:

First, would including a fourth disk be as simple as it looks? I've got 7.2's ZFS spanning 3 disks at the moment, but I've got a fourth sitting around and figure that it might as well be in the server.

It will fit well on 4 disks, but it will require to recreate the ZFS pool, you will just create raidz over 4 disks.

Quote:

Second, do you have any recommendations for maintaining data integrity during the move? I've got external HDD's that can hold all of my stuff, but they're just FAT32 and lack the kind of checksum protection that ZFS is giving me on the current system. My plan would be to use all three of the current disks and a fourth in the new system, but that will require wiping out the current filesystem.

You can tar(1) and split(1) all your data into that fat32 filesystem (parts need to be smaller then 4GB), You may as well create TWO copies of your data there, in two folders, or just make UFS filesystem there.

dewarrn1 · #9 April 4th, 2010, 22:24

Very cool, I'll get that process underway. I'll probably generate some par2 data for those split tar files as well, just in case. Thanks!

dewarrn1 · #10 April 9th, 2010, 05:46

This worked almost exactly as advertised! I ended up with 4x500GB HDDs with a 4-way mirrored base system, 4 GB swap spread across the disks, and ~1.3TB ZFS. My only hiccup was the "zpool import -D" bit, which for some reason didn't want to play nice. However, "zpool import basefs" did the trick, and now I'm getting things back onto ZFS. Nice work, V!

vermaden · #11 April 9th, 2010, 08:26

Quote:

Originally Posted by dewarrn1

This worked almost exactly as advertised! I ended up with 4x500GB HDDs with a 4-way mirrored base system, 4 GB swap spread across the disks, and ~1.3TB ZFS. My only hiccup was the "zpool import -D" bit, which for some reason didn't want to play nice. However, "zpool import basefs" did the trick, and now I'm getting things back onto ZFS. Nice work, V!

Good to know that it also works for others

The first version included zpool import basefs, but after messing with 3 disks it imported with zpool import -D so I changed the formula, I think I will include both just in case, thanks.

Kami · #12 April 17th, 2010, 17:55

I've used your guide and had the same problem as dewarrn1 when trying to import the pool ( zpool import -D), but zpool import basefs did the trick here as well

Oh, I installed the system on 2 drives and used raid1 for all the slices (/, /usr and /var).

I plan to add another disk in the future and I will keep you updated how the process of adding another disk to the zfs pool goes.

Anyway, great guide.

vermaden · #13 April 18th, 2010, 14:11

Quote:

Originally Posted by Kami

I plan to add another disk in the future and I will keep you updated how the process of adding another disk to the zfs pool goes.

But remember that You would have to destroy the current mirror and then create RAIDZ for example.

Quote:

Originally Posted by Kami

Anyway, great guide.

Thanks mate.

mefizto · #14 May 3rd, 2010, 02:06

Dear vermaden or anybody,

would it be possible/prudent to modify the installation to have the entire / (USB) and /usr (ZFS) on one (mirrored) disk, in particular flash and having the rest of the file system, i.e., SWAP, /tmp, /var, /home, etc., on RAIDed hard drives using ZFS?

The motivation would be to further separate the OS/Application (fairly static on my system) from the data.

Thank you,

M

graudeejs · #15 May 3rd, 2010, 09:33

Quote:

Originally Posted by mefizto

Dear vermaden or anybody,

would it be possible/prudent to modify the installation to have the entire / (USB) and /usr (ZFS) on one (mirrored) disk, in particular flash and having the rest of the file system, i.e., SWAP, /tmp, /var, /home, etc., on RAIDed hard drives using ZFS?

The motivation would be to further separate the OS/Application (fairly static on my system) from the data.

Thank you,

M

I don't see any reason why this couldn't be done.

P.S.
Are you from Latvian Linux center?

vermaden · #16 May 3rd, 2010, 10:12

@mefizto

If You want both / and /usr on separate disks/USB then it would be better, to create RAID 1 with gmirror on that USB drives, and use UFS for /usr on that USB disk, and then put all other filesystems with swap on remaining harddisks.

mefizto · #17 May 3rd, 2010, 22:48

Dear killasmurf86,

thank you for the reply. And, no, I am not from Latvian Linux center.

Dear vermaden,

Code:

. . .it would be better, to create RAID 1 with gmirror on that USB drives, . . .

That was what I meant by "to have the entire / (USB) and /usr (ZFS) on one (mirrored) disk". Sorry for my imprecise English.

Code:

. . .use UFS for /usr on that USB disk. . .

What would be the advantage of using UFS instead of ZFS for /usr?

If it is not much bother, could you indicate, at least in general terms, which parts of your procedure have to be changed?

Thank you,

M

vermaden · #18 May 3rd, 2010, 23:27

Quote:

What would be the advantage of using UFS instead of ZFS for /usr?

If it is not much bother, could you indicate, at least in general terms, which parts of your procedure have to be changed?

Keep small / and /usr (base system) on UFS (in case any problems with ZFS) to have fully working 'repair' environment, put all the rest on ZFS tank.

About changes, You would have to create additional UFS partition (e) for /usr of course in section 1.4:

Code:

# cat > label << __EOF__
# /dev/ad0s1:
8 partitions:
  a: 512m  0 4.2BSD
  e: *     * 4.2BSD
__EOF__

mefizto · #19 May 4th, 2010, 02:14

Dear vermaden,

thank you for your help.

Kindest regards,

M

vermaden · #20 May 4th, 2010, 05:59

@mefizto

You are welcome mate.

Daisuke_Aramaki · #21 May 11th, 2010, 12:36

Great guide vermaden. Worked out perfect setting the system bottom up on a new laptop.

vermaden · #22 May 11th, 2010, 13:12

@Daisuke_Aramaki

Thanks 'n' Welcome

mystique · #23 June 1st, 2010, 13:42

Code:

DISKS: 3                                            
# cat > part << __EOF__                               
p 1 165 63  512M                                     
p 2 165  * 1024M                                      
p 3 159  *     *                                      
p 4   0  0     0                                    
a 1                                                
__EOF__                                               
                                                      
# fdisk -f part ad0                               
# fdisk -f part ad1                               
# fdisk -f part ad2

My disks are ad4, ad6, ad8 and ad10..

what would I change here? because this is not working for me;

I understand fdisk -f part ad4 would be the right command but when I do that I get this error:

Code:

******* Working on device /dev/ad4 *******
fdisk: invalid fdisk partition table found
fdisk: geom not found: "ad4"

thanks in advance..

mystique · #24 June 1st, 2010, 14:17

So it looks like I found that I was not using the full path to add the kernel modules and was getting an error that I over looked.. hence why it could not find ad4; *blush* my bad..

So after that I was able to finish the install but when I get to the 'reboot in single user mode' I have some issues..

I have a usb keyboard and can not use the keyboard at boot time; odd..
when it boots I get the mountroot> prompt; also odd..

but when I look back through dmesg I do not see my (ad4, ad6, ad8, ad10) but rather I see ada0, ada1, ada2, ada3..

I am assuming that has *something* to do with ahci (something I've not used yet)

so what do I do with that?

Code:

# fdisk -f part ada0                                    
# fdisk -f part ada1                               
# fdisk -f part ada2
# fidsk -f part ada3

but do I need to load the ahci module beforehand?

thanks in advance.

zeroseven · #25 June 8th, 2010, 09:21

I may be partially in the wrong place to ask this question. I have a sparc64 system and before I set myself up for a lot of desk to face action, I was wondering if I could accomplish this sort of install using the livefs in combination with disk1?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Minimal install of Xorg - howto?	cpcnw	X.Org	26	April 22nd, 2012 06:23
HOWTO: Modern FreeBSD Install (vermaden way)	vermaden	Howtos & FAQs (Moderated)	92	February 8th, 2011 23:03
howto install freeBSD on iMac/howto boot from CD	leizhenhua	Other Architectures	1	June 15th, 2009 06:09
HOWTO: Install and setup MLDonkey on FreeBSD	tangram	Howtos & FAQs (Moderated)	0	November 17th, 2008 11:30

The Following 16 Users Say Thank You to vermaden For This Useful Post:
depayns (February 23rd, 2011), dewarrn1 (April 4th, 2010), edhunter (April 19th, 2010), gnemmi (April 5th, 2010), GoksinAkdeniz (May 9th, 2010), ikbendeman (October 13th, 2010), lme@ (June 2nd, 2010), magistr (April 24th, 2010), nix (October 25th, 2010), overmind (February 16th, 2011), rabfulton (June 28th, 2010), tempo (January 20th, 2012), Unixworld (May 7th, 2011), vand777 (January 29th, 2011), vertexSymphony (June 15th, 2010), zeissoctopus (October 28th, 2011)

The Following 8 Users Say Thank You to vermaden For This Useful Post:
Caliante (April 5th, 2010), edhunter (April 19th, 2010), gnemmi (April 5th, 2010), GoksinAkdeniz (May 9th, 2010), magistr (April 24th, 2010), Unixworld (May 7th, 2011), z0ran (May 2nd, 2011), zeissoctopus (October 28th, 2011)

The Following 5 Users Say Thank You to vermaden For This Useful Post:
gnemmi (April 5th, 2010), GoksinAkdeniz (May 9th, 2010), Unixworld (May 7th, 2011), zeissoctopus (October 28th, 2011), zeroseven (October 22nd, 2010)

The Following User Says Thank You to vermaden For This Useful Post:
dewarrn1 (April 4th, 2010)