HOWTO FREEBSD 8.1 + GELI + ZFS + LARGE DISKS
I encountered this problem for the first time, when I had to install a server with a Raid 5 6Tb config. Every time I installed FreeBSD it just created slices of 1.5 Tb, I wanted to create a small FreeBSD installation and create a big slice (5,5 Tb) for backup purposes. So I went on a search on the web. I wasn't lucky at all, this can mean two things, or I am doing something very wrong or there wasn't a solution yet. So I got creative and came up with the solution you see below. If you have a better way of doing this please let me know.
My setup consists of:
mfid0 5,5 Tb – Will be split in 2 GB Swap 1 GB / and the remaining will be for ZFS.
I will be using a GPT table because of the large disks.
You will have to make a FreeBSD usb installation disk. You can download the usb img here.
And you will need a FreeBSD installation Livefs disc. You can download the installation Livefs disc here.
Installation and setup
The steps:
1. Boot Livefs disk and drop to Fixit shell.
2. Setup partitions/slices on the internal disks.
3. Install FreeBSD.
4. Boot into OS modify partitions copy OS reboot into new OS.
5. Setup GELI on the disk.
6. Create the ZFS pool.
7. Mount ZFS in the right place.
8. Copy OS to ZFS tank.
9. Reboot and done.
Step 1 : Boot Livefs disk and drop to Fixit shell.
First plug in the FreeBSD usb and boot in to the Livefs cd and start the Fixit, (Where using a Livefs disk because not all servers boot the usb disk versions correctly).
Step 2 : Setup partitions/slices on the internal disks.
Create a new Partition table on mfid0 we are using GPT
Create FreeBSD boot partition to contain the boot code.
Write protective MBR to sector 0 of disk
Create swap
Create /
Create /everything
Show partitions
Step 3 : Install FreeBSD 8.1.
Format ufs
Mount Partition
Install FreeBSD
Copy GENERIC kernel
Create new fstab
Step 4 : Boot into OS modify partitions copy OS reboot into new OS.
After the reboot we are going to remove swap
Now we are going to create a 2Gb ufs partition
This will be mfid0p5
Mount mfid0p5
Tar Filesystem
Now reboot and when starting the boot loader press space because we want to boot from different partition
Step 5 : Setup GELI on the disks.
When you are booted in to the new installation we can start making the crypt and zfs tank
Create crypt
Attach disk
Step 6 : Create the ZFS pool.
Create zfs tank
Step 7 : Mount ZFS in the right place.
Format mfid0p3
Mount mfid0p3
Step 8 : Copy OS to ZFS tank.
Copy boot
Create symlink
Edit loader
Edit fstab
We have a working /boot now, so let's fill up the ZFS root
We are now going to restore the swap partition that we earlier removed to use as ufs
We need to update fstab
Final step is to prevent ZFS from re-mounting tank, as it will be used to root filesystem. This can be done by
Step 9 : Reboot
I encountered this problem for the first time, when I had to install a server with a Raid 5 6Tb config. Every time I installed FreeBSD it just created slices of 1.5 Tb, I wanted to create a small FreeBSD installation and create a big slice (5,5 Tb) for backup purposes. So I went on a search on the web. I wasn't lucky at all, this can mean two things, or I am doing something very wrong or there wasn't a solution yet. So I got creative and came up with the solution you see below. If you have a better way of doing this please let me know.
My setup consists of:
mfid0 5,5 Tb – Will be split in 2 GB Swap 1 GB / and the remaining will be for ZFS.
I will be using a GPT table because of the large disks.
- mfid0p3 / 1G
- mfid0p2 swap 2G
- mfid0p4 everything else
You will have to make a FreeBSD usb installation disk. You can download the usb img here.
Code:
dd if=DIR/FreeBSD-8.1-RELEASE-amd64-memstick.img of=/dev/sdb bs=64k
And you will need a FreeBSD installation Livefs disc. You can download the installation Livefs disc here.
Installation and setup
The steps:
1. Boot Livefs disk and drop to Fixit shell.
2. Setup partitions/slices on the internal disks.
3. Install FreeBSD.
4. Boot into OS modify partitions copy OS reboot into new OS.
5. Setup GELI on the disk.
6. Create the ZFS pool.
7. Mount ZFS in the right place.
8. Copy OS to ZFS tank.
9. Reboot and done.
Step 1 : Boot Livefs disk and drop to Fixit shell.
First plug in the FreeBSD usb and boot in to the Livefs cd and start the Fixit, (Where using a Livefs disk because not all servers boot the usb disk versions correctly).
Step 2 : Setup partitions/slices on the internal disks.
Create a new Partition table on mfid0 we are using GPT
Code:
# gpart create -s GPT mfid0
Code:
# gpart add -t freebsd-boot -s 128 mfid0
Code:
# gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptboot -i 1 mfid0
Code:
# gpart -t freebsd-swap -s 2G mfid0
Code:
# gpart -t freebsd-ufs -s 1G mfid0
Code:
# gpart -t freebsd-ufs mfid0
Code:
# gpart show
34 11717836733 mfid0 GPT (5.5T)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (2.0G)
8388770 2097152 3 freebsd-ufs (1.0G)
10485922 11707350845 5 freebsd-ufs (5.5T)
Step 3 : Install FreeBSD 8.1.
Format ufs
Code:
# newfs mfid0p3
# newfs mfid0p4
Mount Partition
Code:
# mount /dev/mfid0p3 /mnt
Install FreeBSD
Code:
# cd /mnt
# mkdir usbstick
# mount /dev/ad0a(FreeBSD usb) /mnt/usbstick
# cd /mnt/usbstick/8.1-RELEASE/base
# ./install.sh
# cd /mnt/usbstick/8.1-RELEASE/kernels
# ./install.sh GENERIC
Copy GENERIC kernel
Code:
# cd /mnt/boot
# rmdir kernel
# cp -Rp GENERIC kernel
Create new fstab
Code:
# vi /mnt/etc/fstab
/dev/mfid0p3 / ufs rw 1 1
# reboot
Step 4 : Boot into OS modify partitions copy OS reboot into new OS.
After the reboot we are going to remove swap
Code:
# gpart delete -i 2 mfid0
Code:
# gpart add -t freebsd-ufs -s 4G mfid0
Mount mfid0p5
Code:
# mount /dev/mfid0p5 /mnt
Code:
# cd /
Code:
# tar cf - --one-file-system * | tar xpf - -C /mnt
Code:
# vi /mnt/etc/fstab
/dev/mfid0p5 / ufs rw 1 1
Code:
# 0:ad(0,p5)/boot/loader
Step 5 : Setup GELI on the disks.
When you are booted in to the new installation we can start making the crypt and zfs tank
Create crypt
Code:
# geli init -b -v -s 4096 /dev/mfid0p4
Code:
# geli attach /dev/mfid0p4
Step 6 : Create the ZFS pool.
Create zfs tank
Code:
# zpool create tank mfid0p4.eli
Step 7 : Mount ZFS in the right place.
Format mfid0p3
Code:
# newfs /dev/mfid0p3
Code:
# mkdir /tank/bootdir
# mount /dev/mfid0p3 /tank/bootdir
Step 8 : Copy OS to ZFS tank.
Copy boot
Code:
# cp -Rp /boot /tank/bootdir/
Code:
# ln -sf /tank/boot /tank/bootdir/boot
Code:
# vi /tank/boot/loader.conf
zfs_load="YES"
geom_eli_load="YES"
Code:
# vi /tank/bootdir/etc/fstab
tank / zfs rw 0 0
Code:
# cd /
# tar cf - --one-file-system * | tar xpf - -C /tank
Code:
# gpart delete -i 5 mfid0
# gpart add -t freebsd-swap -s 4G mfid0
Code:
# vi /tank/etc/fstab
tank / zfs rw 0 0
/dev/mfid0p3 /bootdir ufs rw 1 1
/dev/mfid0p2 swap swap sw 0 0
Code:
# zfs set mountpoint=legacy tank
Step 9 : Reboot
Code:
# reboot