FreeBSD 9 ZFS the easy way

srivo

Member

Reaction score: 6
Messages: 65

Here is how I install FreeBSD 9 with ZFS. This How-To is strongly inspired by gkontos "Root On ZFS @ FreeBSD 9". I will make it a little easier by using the installer to install the base system and setup time, network and everything else.

I strongly suggest you take a look at gkontos "Root On ZFS @ FreeBSD9". He go further into optimizing zfs partition. I will keep it simple here.
  1. Boot from a FreeBSD9 installation CD or DVD and choose install
  2. Select the keyboard you like
  3. Type the name you want for that machine
  4. Select what you want from the distribution. You can select doc, ports, source if you like
  5. On the partitioning screen, select Shell
  6. Now we will create the ZFS partition:
    Code:
    gpart create -s gpt ada0
    gpart add -b 34 -s 94 -t freebsd-boot ada0
    gpart add -t freebsd-zfs -l disk0 ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
    
  7. We will create the zfs pool mount on /mnt:
    Code:
    zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
    zpool export zroot
    zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
    zpool set bootfs=zroot zroot
    
  8. Copy the zpool.cache file on the disk to be able to get it back after installation:
    Code:
    cp /var/tmp/zpool.cache /mnt/zpool.cache
    
  9. Add swap. In that case I add 4GB of swap:
    Code:
    zfs create -V 4G zroot/swap
    zfs set org.freebsd:swap=on zroot/swap
    
  10. Type “exit” and go back to the let the installer finish the job
  11. Type a password for root
  12. Choose your network card and configure your network
  13. Set clock and time zone
  14. Select system configuration like sshd...
  15. Configure dumdev if you like
  16. Add other user if you like
  17. Select exit to finish installation
  18. On the manual configation screen choose YES, you need to copy back the zpool.cache file (very important step):
    Code:
    cp /zpool.cache /boot/zfs/zpool.cache
    
  19. add the zfs driver:
    Code:
    echo 'zfs_enable="YES"' >> /etc/rc.conf
    echo 'zfs_load="YES"' >> /boot/loader.conf
    echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
    
  20. Type exit and choose reboot
  21. After reboot, it's now a good idea to take your first snapshot
    Code:
    zfs snapshot zroot@config
    
You now have a working FreeBSD ZFS system and a first snapshot to rollback to in case of emergency.
 
OP
S

srivo

Member

Reaction score: 6
Messages: 65

If the system doesn't boot!

Don't panic! You probably forgot one operation.

Boot back with the installation CD and choose "live CD" and import back your zpool:
Code:
zpool import -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

You will now have access to your zfs file system on /mnt

These is basicaly two pitfall, you forgot to copy the zpool.cache file or you forgot to load the zfs drivers.

Copy back the zpool.cache file
Code:
cp /var/tmp/zpool.cahe /boot/zfs/zpool.cache


Check back your /etc/rc.conf file to see if
Code:
zfs_enable="YES"

Is present.

Check back the /boot/loader.conf file to see if
Code:
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

are present.
 

TheHolm

New Member


Messages: 3

Thank you for excellent guide. Works for me.
Is there any particular reason why you put swap on ZFS instead of bare GPT partition?
 

ShelLuser

Son of Beastie

Reaction score: 1,830
Messages: 3,622

FILE tags, and it's "FreeBSD"

First a quick thanks for posting, even after one year extremely helpful. One small addition if I may..

srivo said:
[*]We will create the zfs pool mount on /mnt:
Code:
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
zpool set bootfs=zroot zroot
This is very good but a downside for me was that one huge ZFS filesystem can be hard to manage later; for example if you decide that you want /var on a separate filesystem. Not a problem on a separate PC, it becomes an issue if you're on a VPS or a remote server (with KVM access yet no peripheral access for example).

So, after I set up the main pool I wanted /var, /usr/ports, /usr/local, /home as well as some sub directories (/var/tmp) to be separated.

To that end I mixed directions of your tutorial with those from the main FBSD FreeBSD manual, and added my own touch (I didn't want /usr to be separate), hope this can help others as well:

Code:
(before the above 'zpool export zroot')

mkdir /mnt/usr
zfs create -o compression=lzjb -o setuid=off -o mountpoint=/usr/ports zroot/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/packages

zfs create -o mountpoint=/usr/local zroot/local

zfs create zroot/var
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/mail
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp

zfs create zroot/home
zfs create zroot/tmp
chmod 1777 /mnt/tmp


The twist should be obvious; by pointing the mount point for zroot/ports to /usr/ports it will still initially end up on /mnt/usr/ports because /mnt is the root for the zroot filesystem, which sits at the top of the hierarchy. After the installation is done otoh, zroot will end up on / (obviously) and thus automatically make sure that your ports filesystem also ends up on the right place (/usr/ports).

The reason I like this so far (disclaimer: with plenty of Unix experience but little specific FreeBSD experience at the time of writing) is this separates my user data (/usr/local, /var, /home) and additional or dynamic data (/usr/ports) from the system data (/, /usr, etc.).

Hope this can give others some ideas as well.
 
OP
S

srivo

Member

Reaction score: 6
Messages: 65

I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.

But your additions are very good. PC-BSD just does them without having to type anything!
 

JakkFrosted

Member

Reaction score: 3
Messages: 25

Thanks for this tutorial. Only thing I had to do differently was use da0 instead of ada0. I'm running a RAID5 using a HP Smart Array card. I was able to find out the device by poking around in /dev with fdisk. It was my second guess at which device was actually the disk, I figured da for "disk array" :e. I'm wondering if there's a faster way to find the disk device names?
 

kjs3

New Member

Reaction score: 1
Messages: 4

JakkFrosted said:
I'm running a RAID5 using a HP Smart Array card.

You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.
 

ShelLuser

Son of Beastie

Reaction score: 1,830
Messages: 3,622

Be warned when using FreeBSD 9.2!

Hi gang,

This is one of my favourite ZFS tutorials (also because I applied the steps above on all my servers) and when playing with the upcoming 9.2 version I noticed that zpool behaved differently:

Code:
# zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
cannot mount '/mnt/zroot': failed to create mountpoint

Although this looked weird to me I could also come up with a theory as to why this behaviour was changed. But taking no chances I eventually decided to write a problem report where it was confirmed that this change in behaviour is indeed intended.

So, if you like the tutorial above and are planning to use FreeBSD 9.2 (or higher) then please keep this in mind. You now need to use this:

# zpool create -m / -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.

If you already used the previously mentioned zpool command and are now wondering why things don't work (and how to fix this): You don't have to start all over again, simply use this command:

# zfs set mountpoint=/ zroot.

This will make sure that your newly created zroot ZFS pool is now mounted under /mnt instead of /mnt/zroot.

Hope this can help some people too.

Edit: null-edit & fixed {cmd} tags gone awol afterwards.
 

kpa

Beastie's Twin

Reaction score: 1,828
Messages: 6,318

[USER=38324]@ShelLuser[/USER]. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.
 
Last edited by a moderator:

ShelLuser

Son of Beastie

Reaction score: 1,830
Messages: 3,622

kpa said:
[USER=38324]@ShelLuser[/USER]. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.
You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.
 
Last edited by a moderator:

kpa

Beastie's Twin

Reaction score: 1,828
Messages: 6,318

ShelLuser said:
You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.

I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.
 

ShelLuser

Son of Beastie

Reaction score: 1,830
Messages: 3,622

kpa said:
I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.
Yeah, but that's not what you want here. The reason to use /mnt is to allow the installer to extract the base system right onto your newly created ZFS structure.
 

JakkFrosted

Member

Reaction score: 3
Messages: 25

kjs3 said:
You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.

Well, it's not a soft RAID, though. Anyone else ever heard of this?
 

kpa

Beastie's Twin

Reaction score: 1,828
Messages: 6,318

JakkFrosted said:
Well, it's not a soft RAID, though. Anyone else ever heard of this?

The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.
 

JakkFrosted

Member

Reaction score: 3
Messages: 25

kpa said:
The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.


Here's where I'm getting confused: Is it that having it over a RAID is redundant, or, is it that there are going to be undesirable issues?
 

JakkFrosted

Member

Reaction score: 3
Messages: 25

J65nko said:

Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup may preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.
 

kjs3

New Member

Reaction score: 1
Messages: 4

JakkFrosted said:
Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup may preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.

Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do its job. Because when you come back with "ZFS screwed up my RAID 5 array data", we're not going to be a bit sympathetic.
 

kjs3

New Member

Reaction score: 1
Messages: 4

srivo said:
I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.

But your additions are very good. PC-BSD just does them without having to type anything!

As long as you have one disk. If you want to do a ZFS install over two or more (like /var or /home on a separate disk), I can't figure a way to make the PC-BSD GUI to do it. So it's precisely no better than using this procedure.
 

wblock@

Beastie Himself
Developer

Reaction score: 3,695
Messages: 13,851

The hardware RAID card also adds another point of hardware that can fail. And if that card fails, you may need an identical one to be able to access the RAID.
 

JakkFrosted

Member

Reaction score: 3
Messages: 25

kjs3 said:
Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do it's job. Because when you come back with "ZFS screwed up my RAID-5 array data", we're not going to be a bit sympathetic.

How frequent of a problem is data corruption? I have never had to deal with it in my 7 years of server adminning. I host simple websites on my servers and a few other server programs here and there. The main reason I went with ZFS was because of the improvements on speed I read about, and, that I like to try new things. I would be willing to migrate the OS install over to a JBOD array with ZFS, but I'm not sure how to go about that. One way I'm thinking is, make a snapshot, download it, make a JBOD with ZFS, then image the array with the snapshot -- but will that snapshot work seamlessly, being that it's moving from a RAID 5 to a JBOD where ZFS is more in control. After all, you are talking as if I should heed these warnings and change things over, and I am willing to do so if I don't have to reinstall the OS from scratch and everything.

wblock@ said:
The hardware RAID card also adds another point of hardware that can fail. And if that card fails, you may need an identical one to be able to access the RAID.

I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.
 

wblock@

Beastie Himself
Developer

Reaction score: 3,695
Messages: 13,851

JakkFrosted said:
I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.

Plain controllers do not put proprietary RAID metadata on the drives. A RAID-Z pool can be moved to another machine with different controllers. Arrays created with RAID controllers have metadata that may or may not work in a different machine when the original one fails.
 
Top