HOWTO: FreeBSD ZFS Madness

Savagedlight · Apr 26, 2013

Is it ok if I reference this howto in a guide I'm currently writing? I'm not going to follow it exactly, but the ZFS setup is heavily inspired by this guide.

vermaden · Apr 26, 2013

@Savagedlight
Sure, go ahead.

xy16644 · Apr 26, 2013

vermaden said:
@xy16644

Show me output of zfs list and mount commands.

Hi @vermaden, here's the requested output:

zfs list:

Code:

NAME         USED  AVAIL  REFER  MOUNTPOINT
zroot       5.26G  14.3G  4.23G  /
zroot/swap  1.03G  15.3G   108K  -

mount:

Code:

zroot on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)

This output is from a test FreeBSD 9.1 amd64 VM. I'm happy to experiment and make any changes in this VM!

Thanks!

vermaden · Apr 27, 2013

@xy16644

Then you need to do these changes to use beadm:

Code:

# zfs snapshot zroot@beadm
# zfs create zroot/ROOT
# zfs send zroot@beadm | zfs receive zroot/ROOT/default

Now edit /zroot/ROOT/default/boot/loader.conf and put there vfs.root.mountfrom="zfs:zroot/ROOT/default" line instead of the existing vfs.root.mountfrom one.

Code:

# zpool set bootfs=zroot/ROOT/default zroot
# reboot

If it works, then the beadm command should work properly now and you can delete everything in the /zroot directory except the ROOT directory, the zroot@beadm snapshot will also not be needed anymore.

xy16644 · Apr 28, 2013

vermaden said:
@xy16644

Then you need to do these changes to use beadm:

Code:

# zfs snapshot zroot@beadm # zfs create zroot/ROOT # zfs send zroot@beadm | zfs receive zroot/ROOT/default

Now edit /zroot/ROOT/default/boot/loader.conf and put there vfs.root.mountfrom="zfs:zroot/ROOT/default" line instead of the existing vfs.root.mountfrom one.

Code:

# zpool set bootfs=zroot/ROOT/default zroot # reboot

If it works, then the beadm command should work properly now and you can delete everything in the /zroot directory except the ROOT directory, the zroot@beadm snapshot will also not be needed anymore.

Thanks very much for the reply! I seem to be getting stuck though...

I ran the following just fine:
zfs snapshot zroot@beadm
zfs create zroot/ROOT
zfs send zroot@beadm | zfs receive zroot/ROOT/default

My first problem is that the following path does not exist on my test system: /zroot/ROOT/default/boot/loader.conf. But I was able to edit the following path instead: /ROOT/default/boot/loader.conf. And I added the following to it:

Code:

vfs.root.mountfrom="zfs:zroot/ROOT/default"

I then ran the following fine:
zpool set bootfs=zroot/ROOT/default zroot

Funny thing is running beadm list gives me the following (BEFORE rebooting):

Code:

BE      Active Mountpoint     Space Created
default R      /ROOT/default   4.2G 2013-04-28 20:54

But after I reboot I can't get a login prompt and get many errors about paths being incorrect (see attached error). It looks like it has half booted up as I don't get a logon prompt.

Any ideas? :stud

PS: I should mention that I don't have a /zroot folder.

Savagedlight · May 3, 2013

Is it possible to have child data sets of the BEADM data set, and have it mount only when its parent is used?

One of many use cases for this would be to store /usr/src with compression enabled, while not sharing it between BEADM data sets, as that would make little sense.

rawthey · May 3, 2013

Savagedlight said:
Is it possible to have child data sets of the BEADM data set, and have it mount only when its parent is used?

One of many use cases for this would be to store /usr/src with compression enabled, while not sharing it between BEADM data sets, as that would make little sense.

Yes, this is what I have:

Code:

curlew:/home/mike% zfs list -o name,compression,mountpoint -r sys/ROOT/kde4.10a sys/DATA sys/NOBACKUP
NAME                              COMPRESS  MOUNTPOINT
sys/DATA                               off  none
sys/DATA/home                         gzip  /home
sys/DATA/home/camera                   off  /home/camera
sys/DATA/home/db                       off  /home/db
sys/DATA/home/photos                   off  /home/photos
sys/DATA/root                          off  /root
sys/DATA/var                           off  none
sys/DATA/var/log                        on  /var/log
sys/NOBACKUP                           off  none
sys/NOBACKUP/nobackup                  off  /nobackup
sys/NOBACKUP/usr                       off  none
sys/NOBACKUP/usr/ports                 off  none
sys/NOBACKUP/usr/ports/distfiles       off  /usr/ports/distfiles
sys/NOBACKUP/usr/ports/packages        off  /usr/ports/packages
sys/ROOT/kde4.10a                      off  legacy
sys/ROOT/kde4.10a/tmp                   on  /tmp
sys/ROOT/kde4.10a/usr                   on  /usr
sys/ROOT/kde4.10a/usr/ports         gzip-9  /usr/ports
sys/ROOT/kde4.10a/usr/src           gzip-9  /usr/src
sys/ROOT/kde4.10a/var                  off  /var
sys/ROOT/kde4.10a/var/db               off  /var/db
sys/ROOT/kde4.10a/var/db/pkg            on  /var/db/pkg
sys/ROOT/kde4.10a/var/empty            off  /var/empty
sys/ROOT/kde4.10a/var/mail              on  /var/mail
sys/ROOT/kde4.10a/var/run              off  /var/run
sys/ROOT/kde4.10a/var/tmp               on  /var/tmp

Each BE has it's own version of /usr/src and /usr/ports. When I create a new BE I want to continue to use my existing distfiles for ports so /usr/ports/distfiles are mounted from sys/NOBACKUP which is outside the BE filesystems, as are various other directories like /home. Note that for this to work some filesystems like sys/NOBACKUP/usr and sys/NOBACKUP/usr/ports need to have their mountpoint set to none

Savagedlight · May 3, 2013

rawthey said:
Yes, this is what I have:
Each BE has it's own version of /usr/src and /usr/ports. When I create a new BE I want to continue to use my existing distfiles for ports so /usr/ports/distfiles are mounted from sys/NOBACKUP which is outside the BE filesystems, as are various other directories like /home. Note that for this to work some filesystems like sys/NOBACKUP/usr and sys/NOBACKUP/usr/ports need to have their mountpoint set to none

Interesting. I would have thought that sys/ROOT/kde4.10a/var and sys/ROOT/somethingelse/var would clash when both have a mountpoint of /var; Does beadm() do some "magic" to the child data sets to decide which ones are automounted?

vermaden · May 3, 2013

Savagedlight said:
Does beadm() do some "magic" to the child data sets to decide which ones are automounted?

It uses canmount=on|off depends on if this BE is active or not.

xy16644 · May 23, 2013

@vermaden,

I'm still having the problem mentioned in post #130. Any ideas?

vermaden · May 24, 2013

xy16644 said:
@vermaden,

I'm still having the problem mentioned in post #130. Any ideas?

Show me output of the zfs get -r mountpoint zroot command.

xy16644 · May 25, 2013

As requested

zfs get -r mountpoint zroot

Code:

NAME                      PROPERTY    VALUE          SOURCE
zroot                     mountpoint  /              local
zroot@beadm               mountpoint  -              -
zroot/ROOT                mountpoint  /ROOT          inherited from zroot
zroot/ROOT/default        mountpoint  /ROOT/default  inherited from zroot
zroot/ROOT/default@beadm  mountpoint  -              -
zroot/swap                mountpoint  -              -

vermaden · May 26, 2013

Try that and reboot.
# zfs set mountpoint=none zroot
# zfs set mountpoint=none zroot/ROOT
# zfs set mountpoint=legacy zroot/ROOT/default

xy16644 · May 26, 2013

vermaden said:
Try that and reboot.
# zfs set mountpoint=none zroot
# zfs set mountpoint=none zroot/ROOT
# zfs set mountpoint=legacy zroot/ROOT/default

Running zfs set mountpoint=none zroot gave me

Code:

cannot unmount '/': Invalid argument

The second and third command ran fine. When I rebooted after running the commands I get the same error as described in my previous post. Any other ideas?

Thanks for your help!

vermaden · May 26, 2013

Do that using live USB or live CD.

xy16644 · May 27, 2013

vermaden said:
Do that using live USB or live CD.

I'm not. FreeBSD is installed in a VM, but it's not a live CD/USB.

vermaden · May 27, 2013

xy16644 said:
I'm not. FreeBSD is installed in a VM but its not a live CD/USB.

If it's VM, then it's even easier. Just download the DVD ISO image and pass it into the virtual machine and boot from it, just like in a 'real' hardware.

Dies_Irae · Jun 3, 2013

vermaden said:
Try that and reboot.
# zfs set mountpoint=none zroot
# zfs set mountpoint=none zroot/ROOT
# zfs set mountpoint=legacy zroot/ROOT/default

@vermaden, thank you very much for this great guide!

On my home PC I finally decided to switch from using UFS to ZFS, so I used your guide as a base (with some modifications), but there is something still unclear to me.

In section "3.1. Server with Two Disks" you created an empty fstab and set the mountpoint of the root filesystem to legacy, but from what is written in zfs(8):

If a file system's mount point is set to legacy, ZFS makes no attempt to manage the file system, and the administrator is responsible for mounting and unmounting the file system.

I expect to have to insert a line in fstab for the root filesystem, but apparently this is not needed.

It seems that # zpool set bootfs=sys/ROOT/default sys implies to ignore the mountpoint option.

So how does this work?

vermaden · Jun 4, 2013

@Dies_Irae,

Welcome.

Dies_Irae said:
In section "3.1. Server with Two Disks" you created an empty fstab and set the mountpoint of the root filesystem to legacy

The / has to be mounted anyway, so that property is not important here.

Dies_Irae said:
I expect to have to insert a line in fstab for the root filesystem, but apparently this is not needed.

The only things needed are

Code:

zfs_enable=YES

in /etc/rc.conf and

Code:

zfs_load=YES

in the /boot/loader.conf file.

Dies_Irae said:
It seems that # zpool set bootfs=sys/ROOT/default sys implies to ignore the mountpoint option.

Yep.

Dies_Irae said:
So how does this work?

Generally the option 'legacy' is just the option that I used for the first time for Boot Environments to distinguish the Boot Environment from other ZFS datasets, it has no other function.

Dies_Irae · Jun 4, 2013

vermaden said:
The / has to be mounted anyway, so that property is not important here.

(...)

Generally the option 'legacy' is just the option that I used for the first time for Boot Environments to distinguish the Boot Environment from other ZFS datasets, it has no other function.

Crystal clear as always. Again, thank you very much! I owe you a beer :beergrin

vermaden · Jun 4, 2013

:beergrin Na zdrowie!*

*Works better for Vodka then Beer

overmind · Jun 5, 2013

xy16644 said:
Running zfs set mountpoint=none zroot gave me

Code:

cannot unmount '/': Invalid argument

The second and third command ran fine. When I rebooted after running the commands I get the same error as described in my previous post. Any other ideas?

Thanks for your help!

I've used @vermaden's tutorial to install FreeBSD on my work machine. Then I wanted to move my work machine to a VirtualBOX Guest FreeBSD (on a Mac). I've used the same installation tutorial on a VirtualBox machine to get a very basic installation of FreeBSD (where I wanted to put my work machine).

But when I've tried to move the data from my work machine to that minimal installation using zfs send/receive command over ssh I got the same error:

Code:

cannot unmount '/': Invalid argument

So to fix the problem on my VirtualBox minimal install I've renamed tank/ROOT to tank/ROOT2. After that I was able to send via ssh to VirtualBox my tank/ROOT and all volumes under tank/ROOT.

When sending ZFS data over ssh I first got the following error:

Code:

cannot umount "operation not permitted"

I had to install sudo on destination machine and add the following lines to /etc/sudoers:

Code:

Cmnd_Alias ZFS = /sbin/zfs

john ALL = NOPASSWD: ZFS

The complete command for sending the volume using zfs send/receive command was:

Code:

zfs send -R tank/ROOT@2013-06-01 | ssh john@10.0.0.127 sudo zfs recv -Fduv tank

overmind · Jun 5, 2013

@vermaden,

A quick question: on your tutorial at section Road Warrior Laptop, you run the following commands:

zpool create -f -o cachefile=/tmp/zpool.cache sys /dev/gpt/sys0
zpool create -f -o cachefile=/tmp/zpool.cache local /dev/gpt/local0.eli
cp /tmp/zpool.cache /mnt/boot/zfs/

At the end the cache file created for local is copied to /boot and not the file for sys pool. My question is in order to boot, I should not create the zpool.cache for local pool, right? Just the one for sys pool?

Since the zpool.cache is only for booting, right?

vermaden · Jun 5, 2013

@overmind,

You were probably hit by that BUG in 9.1-RELEASE:
http://lists.freebsd.org/pipermail/freebsd-bugs/2012-May/048757.html

Its fixed in STABLE, but not in RELEASE, well 8.4-RELEASE also has it fixed.

overmind · Jun 5, 2013

vermaden said:
@overmind,

You were probably hit by that BUG in 9.1-RELEASE:
http://lists.freebsd.org/pipermail/freebsd-bugs/2012-May/048757.html

Its fixed in STABLE, but not in RELEASE, well 8.4-RELEASE also has it fixed.

Thank you! I think that's my case and yes, I have 9.1-RELEASE. I thought it's a feature not being able to overwrite pool/volume that is mounted as root.

HOWTO: FreeBSD ZFS Madness

Attachments