FreeBSD 9 ZFS the easy way

Here is how I install FreeBSD 9 with ZFS. This How-To is strongly inspired by gkontos "Root On ZFS @ FreeBSD 9". I will make it a little easier by using the installer to install the base system and setup time, network and everything else.

I strongly suggest you take a look at gkontos "Root On ZFS @ FreeBSD9". He go further into optimizing zfs partition. I will keep it simple here.
  1. Boot from a FreeBSD9 installation CD or DVD and choose install
  2. Select the keyboard you like
  3. Type the name you want for that machine
  4. Select what you want from the distribution. You can select doc, ports, source if you like
  5. On the partitioning screen, select Shell
  6. Now we will create the ZFS partition:
    Code:
    gpart create -s gpt ada0
    gpart add -b 34 -s 94 -t freebsd-boot ada0
    gpart add -t freebsd-zfs -l disk0 ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
  7. We will create the zfs pool mount on /mnt:
    Code:
    zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
    zpool export zroot
    zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
    zpool set bootfs=zroot zroot
  8. Copy the zpool.cache file on the disk to be able to get it back after installation:
    Code:
    cp /var/tmp/zpool.cache /mnt/zpool.cache
  9. Add swap. In that case I add 4GB of swap:
    Code:
    zfs create -V 4G zroot/swap
    zfs set org.freebsd:swap=on zroot/swap
  10. Type “exit” and go back to the let the installer finish the job
  11. Type a password for root
  12. Choose your network card and configure your network
  13. Set clock and time zone
  14. Select system configuration like sshd...
  15. Configure dumdev if you like
  16. Add other user if you like
  17. Select exit to finish installation
  18. On the manual configation screen choose YES, you need to copy back the zpool.cache file (very important step):
    Code:
    cp /zpool.cache /boot/zfs/zpool.cache
  19. add the zfs driver:
    Code:
    echo 'zfs_enable="YES"' >> /etc/rc.conf
    echo 'zfs_load="YES"' >> /boot/loader.conf
    echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
  20. Type exit and choose reboot
  21. After reboot, it's now a good idea to take your first snapshot
    Code:
    zfs snapshot zroot@config
You now have a working FreeBSD ZFS system and a first snapshot to rollback to in case of emergency.
 
If the system doesn't boot!

Don't panic! You probably forgot one operation.

Boot back with the installation CD and choose "live CD" and import back your zpool:
Code:
zpool import -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
You will now have access to your zfs file system on /mnt

These is basicaly two pitfall, you forgot to copy the zpool.cache file or you forgot to load the zfs drivers.

Copy back the zpool.cache file
Code:
cp /var/tmp/zpool.cahe /boot/zfs/zpool.cache

Check back your /etc/rc.conf file to see if
Code:
zfs_enable="YES"
Is present.

Check back the /boot/loader.conf file to see if
Code:
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
are present.
 
Thank you for excellent guide. Works for me.
Is there any particular reason why you put swap on ZFS instead of bare GPT partition?
 
FILE tags, and it's "FreeBSD"

First a quick thanks for posting, even after one year extremely helpful. One small addition if I may..

srivo said:
[*]We will create the zfs pool mount on /mnt:
Code:
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
zpool set bootfs=zroot zroot
This is very good but a downside for me was that one huge ZFS filesystem can be hard to manage later; for example if you decide that you want /var on a separate filesystem. Not a problem on a separate PC, it becomes an issue if you're on a VPS or a remote server (with KVM access yet no peripheral access for example).

So, after I set up the main pool I wanted /var, /usr/ports, /usr/local, /home as well as some sub directories (/var/tmp) to be separated.

To that end I mixed directions of your tutorial with those from the main FBSD FreeBSD manual, and added my own touch (I didn't want /usr to be separate), hope this can help others as well:

Code:
(before the above '[I]zpool export zroot[/I]')

mkdir /mnt/usr
zfs create -o compression=lzjb -o setuid=off -o mountpoint=/usr/ports zroot/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/packages

zfs create -o mountpoint=/usr/local zroot/local

zfs create zroot/var
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/mail
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp

zfs create zroot/home
zfs create zroot/tmp
chmod 1777 /mnt/tmp

The twist should be obvious; by pointing the mount point for zroot/ports to /usr/ports it will still initially end up on /mnt/usr/ports because /mnt is the root for the zroot filesystem, which sits at the top of the hierarchy. After the installation is done otoh, zroot will end up on / (obviously) and thus automatically make sure that your ports filesystem also ends up on the right place (/usr/ports).

The reason I like this so far (disclaimer: with plenty of Unix experience but little specific FreeBSD experience at the time of writing) is this separates my user data (/usr/local, /var, /home) and additional or dynamic data (/usr/ports) from the system data (/, /usr, etc.).

Hope this can give others some ideas as well.
 
I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.

But your additions are very good. PC-BSD just does them without having to type anything!
 
Thanks for this tutorial. Only thing I had to do differently was use da0 instead of ada0. I'm running a RAID5 using a HP Smart Array card. I was able to find out the device by poking around in /dev with fdisk. It was my second guess at which device was actually the disk, I figured da for "disk array" :e. I'm wondering if there's a faster way to find the disk device names?
 
JakkFrosted said:
I'm running a RAID5 using a HP Smart Array card.

You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.
 
Be warned when using FreeBSD 9.2!

Hi gang,

This is one of my favourite ZFS tutorials (also because I applied the steps above on all my servers) and when playing with the upcoming 9.2 version I noticed that zpool behaved differently:

Code:
# zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
cannot mount '/mnt/zroot': failed to create mountpoint
Although this looked weird to me I could also come up with a theory as to why this behaviour was changed. But taking no chances I eventually decided to write a problem report where it was confirmed that this change in behaviour is indeed intended.

So, if you like the tutorial above and are planning to use FreeBSD 9.2 (or higher) then please keep this in mind. You now need to use this:

# zpool create -m / -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.

If you already used the previously mentioned zpool command and are now wondering why things don't work (and how to fix this): You don't have to start all over again, simply use this command:

# zfs set mountpoint=/ zroot.

This will make sure that your newly created zroot ZFS pool is now mounted under /mnt instead of /mnt/zroot.

Hope this can help some people too.

Edit: null-edit & fixed {cmd} tags gone awol afterwards.
 
@ShelLuser. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.
 
Last edited by a moderator:
kpa said:
@ShelLuser. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.
You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.
 
Last edited by a moderator:
ShelLuser said:
You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.

I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.
 
kpa said:
I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.
Yeah, but that's not what you want here. The reason to use /mnt is to allow the installer to extract the base system right onto your newly created ZFS structure.
 
kjs3 said:
You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.

Well, it's not a soft RAID, though. Anyone else ever heard of this?
 
JakkFrosted said:
Well, it's not a soft RAID, though. Anyone else ever heard of this?

The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.
 
kpa said:
The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.


Here's where I'm getting confused: Is it that having it over a RAID is redundant, or, is it that there are going to be undesirable issues?
 
J65nko said:

Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup may preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.
 
JakkFrosted said:
Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup may preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.

Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do its job. Because when you come back with "ZFS screwed up my RAID 5 array data", we're not going to be a bit sympathetic.
 
srivo said:
I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.

But your additions are very good. PC-BSD just does them without having to type anything!

As long as you have one disk. If you want to do a ZFS install over two or more (like /var or /home on a separate disk), I can't figure a way to make the PC-BSD GUI to do it. So it's precisely no better than using this procedure.
 
The hardware RAID card also adds another point of hardware that can fail. And if that card fails, you may need an identical one to be able to access the RAID.
 
kjs3 said:
Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do it's job. Because when you come back with "ZFS screwed up my RAID-5 array data", we're not going to be a bit sympathetic.

How frequent of a problem is data corruption? I have never had to deal with it in my 7 years of server adminning. I host simple websites on my servers and a few other server programs here and there. The main reason I went with ZFS was because of the improvements on speed I read about, and, that I like to try new things. I would be willing to migrate the OS install over to a JBOD array with ZFS, but I'm not sure how to go about that. One way I'm thinking is, make a snapshot, download it, make a JBOD with ZFS, then image the array with the snapshot -- but will that snapshot work seamlessly, being that it's moving from a RAID 5 to a JBOD where ZFS is more in control. After all, you are talking as if I should heed these warnings and change things over, and I am willing to do so if I don't have to reinstall the OS from scratch and everything.

wblock@ said:
The hardware RAID card also adds another point of hardware that can fail. And if that card fails, you may need an identical one to be able to access the RAID.

I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.
 
JakkFrosted said:
I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.

Plain controllers do not put proprietary RAID metadata on the drives. A RAID-Z pool can be moved to another machine with different controllers. Arrays created with RAID controllers have metadata that may or may not work in a different machine when the original one fails.
 
Back
Top