HOWTO: FreeBSD ZFS Madness

beadm question

I haven't used beadm before. I think the concept is really cool, and it's great that you've ported it to FreeBSD! That being said, I gave it a try and ran into a few questions.. If anyone might be able to clarify a few things, that would be great!

If I start with no snapshots:

Code:
# zfs list -t snapshot
no datasets available

# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /           597M static 2012-05-18 07:36

Only the default... let's say I want to install updates on my system, so I ...

Code:
# beadm create test
Created successfully

# zfs list -t snapshot
NAME                    USED  AVAIL  REFER  MOUNTPOINT
sys/ROOT/default@test      0      -   597M  -

So far, I think I understand. I've created a snapshot of "default" called "test". It isn't active yet, but in theory, if I activate it, and reboot, I should be using "test". If I want to go back to default, I should be able to re-activate default.

Code:
# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /           597M static 2012-05-18 07:36
test    -      -             1K static 2012-05-22 09:49
Just for (useless) fun, I create another snapshot called "test1":

Code:
# beadm create test1
Created successfully

In theory, test and test1, should be the same since they are both snapshots of default?

Code:
# zfs list -t snapshot
NAME                     USED  AVAIL  REFER  MOUNTPOINT
sys/ROOT/default@test   62.5K      -   597M  -
sys/ROOT/default@test1      0      -   597M  -

I'm fine here, until I decide to activate test1.

Code:
# beadm activate test1
Activated successfully

# zfs list -t snapshot
NAME                   USED  AVAIL  REFER  MOUNTPOINT
sys/ROOT/test1@test   62.5K      -   597M  -
sys/ROOT/test1@test1    19K      -   597M  -
This output, which may very well be correct confuses me as a first time beadm user.
It's not quite clear what happens on activate. I know snapshots are read only, but I wouldn't have expected default@test or default@test1 snapshots to change -- is this correct?

Code:
# beadm list
BE      Active Mountpoint Space Policy Created
default N      /              0 static 2012-05-18 07:36
test    -      -             1K static 2012-05-22 09:49
test1   R      -           597M static 2012-05-22 09:50

Now let's say I want to go back to default -- I would imagine everything would go back to the way it was before:

Code:
# beadm activate default
# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /              0 static 2012-05-18 07:36
test    -      -             1K static 2012-05-22 09:49
test1   -      -           597M static 2012-05-22 09:50

That seemed okay, but ...

Code:
# zfs list -t snapshot
NAME                   USED  AVAIL  REFER  MOUNTPOINT
sys/ROOT/test1@test   62.5K      -   597M  -
sys/ROOT/test1@test1    19K      -   597M  -

Hmmm still not sure I understand that output.

So let's say I want to go back to default and start again.

Code:
# beadm destroy test
Are you sure you want to destroy 'test'?
This action cannot be undone (y/[n]): y
Destroyed successfully

# beadm destroy test1
Are you sure you want to destroy 'test1'?
This action cannot be undone (y/[n]): y
Note: No error message, but no destroyed successfully message either.. hmm.

# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /          97.5K static 2012-05-18 07:36
test1   -      -           597M static 2012-05-22 09:50

So I'll try running destroy again.

Code:
# beadm destroy test1
Are you sure you want to destroy 'test1'?
This action cannot be undone (y/[n]): y

In fact, I can run it over and over again with the same result, but if I do:

# sh -xv beadm destroy test1

I can see in the output:

Code:
zfs promote cannot destroy \''sys/ROOT/test1'\'':' filesystem has dependent clones

and then after a "+ zfs promote sys/ROOT/default", I see:
Code:
+ echo 'Destroyed successfully'
Destroyed successfully
and now:

Code:
# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /           597M static 2012-05-18 07:36
test1   -      -          71.5K static 2012-05-22 10:25

# zfs list -t snapshot
NAME                     USED  AVAIL  REFER  MOUNTPOINT
sys/ROOT/default@test1  63.5K      -   597M  -
The odd thing is that after, I was able to do:

Code:
# beadm destroy test1
Are you sure you want to destroy 'test1'?
This action cannot be undone (y/[n]): y
Destroyed successfully
# beadm list
BE      Active Mountpoint Space Policy Created
default NR     /           597M static 2012-05-18 07:36
I guarantee that some of this is my lack of understanding of beadm, but I suspect there's a bug hiding in there as well.

A few other minor questions:
1) How does one determine which version of beadm that they have? I downloaded mine today from SF, so I imagine it's the latest version as of today, but how do I keep track? (May I recommend a version number at the top of the script?)

2) In your instructions above, after installing FreeBSD with ZFS root, you recommend taking a snapshot. I think that's a great idea, but do I need to use zfs snapshot for that, or if I have beadm, would I just use beadm? That is, would there ever be a circumstance where I would need to take a snapshot of ROOT where beadm wouldn't be the tool to use?

Thanks for any assistance.

Jason.
 
I am having the strangest problem, it keeps coming back!

Code:
Mounting from zfs:sys/ROOT/default failed with error 2

Could anyone help me, please?
 
kangaroo said:
# beadm list
BE Active Mountpoint Space Policy Created
default N / 0 static 2012-05-18 07:36
test - - 1K static 2012-05-22 09:49
test1 R - 597M static 2012-05-22 09:50

Now let's say I want to go back to default -- I would imagine everything would go back to the way it was before:

# beadm activate default
# beadm list
BE Active Mountpoint Space Policy Created
default NR / 0 static 2012-05-18 07:36
test - - 1K static 2012-05-22 09:49
test1 - - 597M static 2012-05-22 09:50

The significant thing to note here is the absence of an Activated successfully message after beadm activate default. This looks like it's caused by the bug (or feature?) of ZFS causing the line zfs set canmount=on ${I} 2> /dev/nul to fail because it's trying to unmount /. There's more background to this in my earlier post in this thread.

Until the canmount issue is resolved you need to reboot into the freshly activated BE before attempting to go back.
 
kangaroo said:
I haven't used beadm before. I think the concept is really cool, and it's great that you've ported it to FreeBSD! That being said, I gave it a try and ran into a few questions.. If anyone might be able to clarify a few things, that would be great!
Thanks ;)

kangaroo said:
So far, I think I understand. I've created a snapshot of "default" called "test". It isn't active yet, but in theory, if I activate it, and reboot, I should be using "test". If I want to go back to default, I should be able to re-activate default.

New BE is creting a snapshot from other BE and then creating a clone from that snapshot, so sys/ROOT/test is a clone of sys/ROOT/default@test snapshot.

kangaroo said:
I'm fine here, until I decide to activate test1.
The activation means zfs promote which diverts the parent/child relationship between sys/ROOT/default and sys/ROOT/test, so after activation sys/ROOT/test will be 'most important' and sys/ROOT/default will be treated as clone of snapshot of sys/ROOT/test (simplifacation) ;)

kangaroo said:
Now let's say I want to go back to default -- I would imagine everything would go back to the way it was before:
There is a BUG in ZFS code which tries to remount a mounted filesystem when changing ZFS canmount property from noauto to on, there is a PR for that and Bryan Drewery is working on getting that one solved. Currently, activating the booted BE is broken because of that - You will activate only the 'main' dataset like sys/ROOT/default, but not sys/ROOT/default/tmp.

kangaroo said:
I guarantee that some of this is my lack of understanding of beadm, but I suspect there's a bug hiding in there as well.
I will look into these issues and try to resolve them, thanks for testing ;)

kangaroo said:
1) How does one determine which version of beadm that they have? I downloaded mine today from SF, so I imagine it's the latest version as of today, but how do I keep track? (May I recommend a version number at the top of the script?)

Just use the GITHUB tag: https://github.com/vermaden/beadm/tree/0.6/

kangaroo said:
2) In your instructions above, after installing FreeBSD with ZFS root, you recommend taking a snapshot. I think that's a great idea, but do I need to use zfs snapshot for that, or if I have beadm, would I just use beadm? That is, would there ever be a circumstance where I would need to take a snapshot of ROOT where beadm wouldn't be the tool to use?
The snapshot is the most important thing, when You have a snapshot, then You can create BE from it with ZFS clone, so creating 'whole' BE is not needed.

kangaroo said:
Thanks for any assistance.
Welcome. Sorry for late response.




donduq said:
I am having the strangest problem, it keeps coming back!

Code:
Mounting from zfs:sys/ROOT/default failed with error 2

Could anyone help me, please?

This error on boot happens when You have BE imported from other machine and
kangaroo said:
s not updated on that BE that You want to boot. Solution is to boot some FreeBSD live CD (FreeBSD ISO or mfsbsd ISO or Frenzy ...) and import pool with -o cachefile=/tmp/zpool.cache, then set mountpoint=/mnt for sys/ROOT/default, then copy that /tmp/zpool.cache file to /mnt/boot/zfs/zpool.cache, then zfs umount -a, then reboot (if I recall correctly).

rawthey said:
Until the canmount issue is resolved you need to reboot into the freshly activated BE before attempting to go back.
Unfortunately, true ;)
 
I'm using only one disk so I'm using the revise step 7 from the 3.1. I'm getting this error
Code:
cannot mount 'sys' : No such file or directory

Can someone please help? Thanks. :)
 
The "cannot mount..." message is normal when booting from the DVD and can be ignored. So, providing the partitions had been created OK in step 6, everything else should continue without any problem.
 
It's a harmless warning, while using the live CD you are not able to create /sys or any other root directory and that is where the warning comes. You can combine steps 7 and 8 into one to omit that warning: [cmd=]# zpool create -f -o cachefile=/tmp/zpool.cache -o mountpoint=none sys mirror /dev/gpt/sys*[/cmd]
 
Can I use beadm to also snapshot encrypted partition (laptop example). Or if I do not have an encrypted partition but a second pool, can I manage that too from beadm?
 
overmind said:
Can I use beadm to also snapshot encrypted partition (laptop example).

Yes, that would be beadm create beName@snapshotNameof that Boot Environment.

Its the same as the zfs snapshot -r sys/ROOT/beName@snapshotName command.

overmind said:
Or if I do not have an encrypted partition but a second pool, can I manage that too from beadm?

I used beadm only to manage 'root' ZFS pool, haven't tried with two ZFS pools.
 
But the encrypted partition is on second pool (called local). And zfs snapshot -r sys/ROOT/beName@snapshotName is for sys pool. So I get:
Code:
Error, cannot create snapshot

I could use another approach to just use a single pool (sys and create a vdev on that pool and encrypt it. Could that be ok or is too much overhead (being zfs on geli on top of zfs)

Do you use laptop example to snapshot encrypted (home) partition too when doing a beadm create ...? The idea is to save states for both partitions.

And thank you, great tutorial and great idea with beadm. This is a dream come true for many sysadmins to ease deployments of servers/updates or for developers ;).
 
Thank you for the nice guide vermaden.

vermaden said:
Code:
gpart add -t freebsd-zfs -l sys${NUMBER} ${I}

Is it helpful to specify the -a or -b switches here for sector alignment?

I will be getting a new laptop soon with an i5-2520M processor that includes "AES New Instructions". With such cryto hardware I'm expecting something around 75% performance with geli versus without. Is this a reasonable expectation?
 
overmind said:
But the encrypted partition is on second pool (called local).

Yes, but beadm does not 'touch' it ;)

overmind said:
And zfs snapshot -r sys/ROOT/beName@snapshotName is for sys pool. So I get:
Code:
Error, cannot create snapshot

I just created such a snapshot, don't know why you can't, try a different/nonexistent snapshot name.

overmind said:
I could use another approach to just use a single pool (sys and create a vdev on that pool and encrypt it. Could that be ok or is too much overhead (being zfs on geli on top of zfs)

It should work ok, but it will be slower because of 'doubled' ZFS.

overmind said:
Do you use laptop example to snapshot encrypted (home) partition too when doing a beadm create ...? The idea is to save states for both partitions.

No, I use beadm only for the sys pool.

If you want, you can create a snapshot of the local pool by yourself with the zfs snapshot command, but it's not needed, we need an installed system state snapshot, not an empty directories snapshot from a local pool ;)

overmind said:
And thank you, great tutorial and great idea with beadm. This is a dream come true for many sysadmins to ease deployments of servers/updates or for developers ;).

The kudos go to OpenSolaris/Solaris developers who created the beadm idea, I just implemented this idea in the FreeBSD world, there is still a PR/PR 167905 that 'blocks' full beadm functionality, so we will have to wait for it being fixed to have a fully working beadm on FreeBSD.




jrm said:
Thank you for the nice guide vermaden.
Welcome.

jrm said:
Is it helpful to specify the -a or -b switches here for sector alignment?
I do not have any 4k drives yet (on purpose ;)) so I can not clarify that.

jrm said:
I will be getting a new laptop soon with an i5-2520M processor that includes "AES New Instructions". With such cryto hardware I'm expecting something around 75% performance with geli versus without. Is this a reasonable expectation?
AESNI will definitely be faster then without, but I don't know exact numbers.
 
Hi, thanks for the great guide. I'm very rusty on FreeBSD, the last version I ran was 4.1. I'm missing the step between 3.1 and 4.0. I have completed 3.1 sucessfully but I'm missing the steps for install to use the pool.

Any pointers would be greatly appreciated.

I am using FreeBSD-9.0-RELEASE-i386-memstick.img as my install medium.

Thanks,
Paul
 
pgrunwald said:
Hi, thanks for the great guide.
Welcome.

pgrunwald said:
I'm missing the step between 3.1 and 4.0. I have completed 3.1 sucessfully but I'm missing the steps for install to use the pool.

Any pointers would be greatly appreciated.
Can you explain more what problem you faced?
 
Hi,

Do I just start the install as normal on the reboot still operating from my USB drive? Going through the install, I was not sure where to install to. I still see ada0 and ada1 with the boot and zfs partitions. I'm just not sure how to proceed with the installation to the pool drive. I have 2 1TB drives and this box will be operating as a NAS via Samba, FTP, and possibly Tahoe-LAFS.

TIA,
Paul
 
pgrunwald said:
Do I just start the install as normal on the reboot still operating from my USB drive?
After booting into live CD/USB and entering the instruction you reboot into just installed FreeBSD, not into the live CD/USB.

pgrunwald said:
Going through the install, I was not sure where to install to.
It's up to you where you want to install it.

pgrunwald said:
I still see ada0 and ada1 with the boot and zfs partitions.
Put command outputs here, I do not exacly know what you mean ;)
 
Thanks, I was confused. I did boot with the memstick out and the computer hangs after the boot prompt at the propeller. Just for grins, I tried this recipe: http://forums.freebsd.org/showthread.php?t=23544 and I get the same results, hang right after the boot prompt.

Motherboard is Atom based Gigabyte GA-D525TUD with 4GB of RAM. I have two Samsung 1TB hard drives on the Intel SATA interface.

Any other suggestions?
 
I have the same board, with 8GB RAM. It seems it can't boot from GPT formatted USB sticks. I have the latest BIOS (F05). I managed to ZFS boot from an MBR formatted USB stick, with non-RAID ZFS on the same stick. It also boots from a mirrored ZFS on two GPT formatted SATA disks.

HTH
 
Ok, I have updated to BIOS version F05, the latest available at the manufacturer. It still hangs at the propeller.

HarryE - thanks for the note! I am able to boot from the FreeBSD-9.0-RELEASE-i386-memstick.img as written by win32diskimager and I'm able to run through this procedure and this one: http://forums.freebsd.org/showthread.php?t=23544 without issue. It just won't come up on the reboot without the memstick.

My drives are set on AHCI.

What next please folks?
 
The problematic PR about ZFS canmount property has been fixed (thanks to Bryan Drewery) and merged to HEAD (with MFC: 1 week): http://freebsd.org/cgi/query-pr.cgi?pr=167905

So now beadm is fully functional on FreeBSD HEAD and will be in 9-STABLE in less then a week, or You may apply the patch Yourself from here: http://freshbsd.org/commit/freebsd/r237119

With these instructions:
[CMD=""]# cd /usr/src/cddl[/CMD]
[CMD=""]# patch -p1 < patch-zfs-canmount[/CMD]
[CMD=""]# make obj depend all install[/CMD]
[CMD=""]# reboot[/CMD]
 
First, thanks for that how-to! It's really helpful to manage servers.

I followed 6.2 to do an upgrade within a jail and it look like something is missing. I got the following error.
Code:
Configuring jails:.
Starting jails:df: /usr/jails/jailed/dev: No such file or directory
mount: /usr/jails/jailed: Not a directory
/etc/rc.d/jail: WARNING: devfs_domount(): Unable to mount devfs on /usr/jails/jailed/dev
/etc/rc.d/jail: WARNING: devfs_mount_jail: devfs was not mounted on /usr/jails/jailed/dev
ln: /usr/jails/jailed/dev/log: No such file or directory
 cannot start jail "jailed":

Like if the zfs jailed volume create by beadm is not mounted.

Also there is a little typo in the man page in the example section it written beadmn instead of beadm.

srivo
 
Back
Top