Howto FreeBSD 8.1 + geli + zfs + large disks

HOWTO FREEBSD 8.1 + GELI + ZFS + LARGE DISKS

I encountered this problem for the first time, when I had to install a server with a Raid 5 6Tb config. Every time I installed FreeBSD it just created slices of 1.5 Tb, I wanted to create a small FreeBSD installation and create a big slice (5,5 Tb) for backup purposes. So I went on a search on the web. I wasn't lucky at all, this can mean two things, or I am doing something very wrong or there wasn't a solution yet. So I got creative and came up with the solution you see below. If you have a better way of doing this please let me know.

My setup consists of:

mfid0 5,5 Tb – Will be split in 2 GB Swap 1 GB / and the remaining will be for ZFS.
I will be using a GPT table because of the large disks.
  • mfid0p3 / 1G
  • mfid0p2 swap 2G
  • mfid0p4 everything else
What are we going to need:

You will have to make a FreeBSD usb installation disk. You can download the usb img here.

Code:
dd if=DIR/FreeBSD-8.1-RELEASE-amd64-memstick.img of=/dev/sdb bs=64k

And you will need a FreeBSD installation Livefs disc. You can download the installation Livefs disc here.

Installation and setup

The steps:
1. Boot Livefs disk and drop to Fixit shell.
2. Setup partitions/slices on the internal disks.
3. Install FreeBSD.
4. Boot into OS modify partitions copy OS reboot into new OS.
5. Setup GELI on the disk.
6. Create the ZFS pool.
7. Mount ZFS in the right place.
8. Copy OS to ZFS tank.
9. Reboot and done.

Step 1 : Boot Livefs disk and drop to Fixit shell.

First plug in the FreeBSD usb and boot in to the Livefs cd and start the Fixit, (Where using a Livefs disk because not all servers boot the usb disk versions correctly).

Step 2 : Setup partitions/slices on the internal disks.

Create a new Partition table on mfid0 we are using GPT
Code:
# gpart create -s GPT mfid0
Create FreeBSD boot partition to contain the boot code.
Code:
# gpart add -t freebsd-boot -s 128 mfid0
Write protective MBR to sector 0 of disk
Code:
# gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptboot -i 1 mfid0
Create swap
Code:
# gpart -t freebsd-swap -s 2G mfid0
Create /
Code:
# gpart -t freebsd-ufs -s 1G mfid0
Create /everything
Code:
# gpart -t freebsd-ufs mfid0
Show partitions

Code:
# gpart show

              34  11717836733  mfid0  GPT  (5.5T)
      34          128      1  freebsd-boot  (64K)
    162      8388608      2  freebsd-swap  (2.0G)
 8388770      2097152      3  freebsd-ufs  (1.0G)
10485922  11707350845      5  freebsd-ufs  (5.5T)

Step 3 : Install FreeBSD 8.1.

Format ufs
Code:
# newfs mfid0p3
# newfs mfid0p4

Mount Partition
Code:
# mount /dev/mfid0p3 /mnt

Install FreeBSD
Code:
# cd /mnt
# mkdir usbstick
# mount /dev/ad0a(FreeBSD usb) /mnt/usbstick
# cd /mnt/usbstick/8.1-RELEASE/base
# ./install.sh
# cd /mnt/usbstick/8.1-RELEASE/kernels
# ./install.sh GENERIC

Copy GENERIC kernel
Code:
# cd /mnt/boot
# rmdir kernel
# cp -Rp GENERIC kernel

Create new fstab
Code:
# vi /mnt/etc/fstab

/dev/mfid0p3       /       ufs        rw          1        1

# reboot

Step 4 : Boot into OS modify partitions copy OS reboot into new OS.

After the reboot we are going to remove swap
Code:
# gpart delete -i 2 mfid0
Now we are going to create a 2Gb ufs partition
Code:
# gpart add -t freebsd-ufs -s 4G mfid0
This will be mfid0p5
Mount mfid0p5
Code:
# mount /dev/mfid0p5 /mnt
Tar Filesystem
Code:
# cd /
Code:
# tar cf - --one-file-system * | tar xpf - -C /mnt
Code:
# vi /mnt/etc/fstab
/dev/mfid0p5       /       ufs        rw          1        1
Now reboot and when starting the boot loader press space because we want to boot from different partition
Code:
# 0:ad(0,p5)/boot/loader

Step 5 : Setup GELI on the disks.

When you are booted in to the new installation we can start making the crypt and zfs tank

Create crypt
Code:
# geli init -b -v -s 4096 /dev/mfid0p4
Attach disk
Code:
# geli attach /dev/mfid0p4

Step 6 : Create the ZFS pool.

Create zfs tank
Code:
# zpool create tank mfid0p4.eli

Step 7 : Mount ZFS in the right place.

Format mfid0p3
Code:
# newfs /dev/mfid0p3
Mount mfid0p3
Code:
# mkdir /tank/bootdir
# mount /dev/mfid0p3 /tank/bootdir

Step 8 : Copy OS to ZFS tank.

Copy boot
Code:
# cp -Rp /boot /tank/bootdir/
Create symlink
Code:
# ln -sf /tank/boot /tank/bootdir/boot
Edit loader
Code:
# vi /tank/boot/loader.conf

zfs_load="YES"
geom_eli_load="YES"
Edit fstab
Code:
# vi /tank/bootdir/etc/fstab

tank      /    zfs     rw     0     0
We have a working /boot now, so let's fill up the ZFS root
Code:
# cd /

# tar cf - --one-file-system * | tar xpf - -C /tank
We are now going to restore the swap partition that we earlier removed to use as ufs
Code:
# gpart delete -i 5 mfid0

# gpart add -t freebsd-swap -s 4G mfid0
We need to update fstab
Code:
# vi /tank/etc/fstab

tank            /           zfs     rw     0     0

/dev/mfid0p3     /bootdir    ufs     rw     1     1

/dev/mfid0p2    swap        swap    sw     0     0
Final step is to prevent ZFS from re-mounting tank, as it will be used to root filesystem. This can be done by
Code:
# zfs set mountpoint=legacy tank

Step 9 : Reboot

Code:
# reboot
 
Excellent guide, but it would be better to export the disks as single drives and let ZFS handle the raid.

Hardware raid and ZFS overlaps and does not make sense (unless you raid several hardware raids).
 
Back
Top