FreeBSD 8.0: '/' on ZFS with compression - how to?

O

omero

Hi there, I'm setting up a new server with 2 disks and an intel i7 920, 8GB ram.

I'm following the mirror tutorial on: http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror

It has worked like charm so far, perfectly.

But, after some workload benchmark test, I noticed i can improve both space and IO throughput by using lzjb by default for the whole ZFS pool, and disabling it for spot places (say, usr/ports/distfiles & packages, etc...).

So i decided to create the zpool 'zroot' using compression=lzjb as default.

But I noticed the kernel cannot be loaded from a ZFS filesystem with compression turned on.

So i left the root filesystem with compression on, but created a new zroot/boot fs turning compression OFF and copied again everythign (to be sure the kernel files are NOT compressed... i know setting properties is valid only for NEW files).

This way I could leave the 'default' to lzjb, but have /boot uncompressed and booting correctly.

But so far no success, the boot gets stuck at

Code: 
FreeBSD/i386 boot
Default: zroot:/boot/kernel/kernel

I even tried from Fixit to change the bootfs properties of zroot from zroot to zroot/boot but didn't work

Code: 
zpool set bootfs=zroot/boot zroot

Do I also have to change
Code: 
vfs.root.mountfrom="zfs:zroot"
to
Code: 
vfs.root.mountfrom="zfs:zroot/boot"
in loader.conf? Or is there something else I can do for making this work.

As i said the goal is:

'root' ZFS filesystem with compression on by default that allows booting.
 
Also tried changing vfs.root.mountfrom to "zfs:zroot/boot"

Same thing, stuck on boot.

Any ideas?
 
SirDice said:
Did you do step 7 (Updating the boot code)?
Click to expand...

Yes. I'm 101% sure I did everything ok, because I'm testing the installation on VMs, and I already SUCCESSFULLY installed ZFS as root (with mirror configuration) more than 10 times.

But if I set compression=lzjb for root filesystem kernel don't load (this is expected I think).

So, I tried to separate /boot into a different filesystem with overriden compression=off properties. But I cannot get this to work, the boot loader doesn't start correctly and hangs.

I think IT IS possible to make it work, but how?
 
To be completely clear.

If i add:

Code: 
zfs set compression=zljb zroot
zfs create -o compression=off zroot/boot
zfs set mountpoint=/boot zroot/boot

to that guide, FreeBSD doesn't start anymore.

How Can I fix this, I mean:

A) Have a 'root' filesystem (i mean the MAIN zfs in the pool, the / one) with compression ON by default
B) Still retain a bootable FreeBSD :)
 
Ah, right.. The key is probably in those last zfs set mountpount commands.
Not sure but you could try # zfs set mountpoint=legacy zroot/boot.
 
I use zfs on root with compression, no mirror though.

Code: 
> zfs get compression zroot
NAME   PROPERTY     VALUE     SOURCE
zroot  compression  lzjb      local
> df -h
Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot                        172G    778M    171G     0%    /
devfs                        1.0K    1.0K      0B   100%    /dev
zroot/tmp                    171G     18M    171G     0%    /tmp
zroot/usr                    222G     51G    171G    23%    /usr
zroot/usr/ports              172G    882M    171G     1%    /usr/ports
zroot/usr/ports/distfiles    172G    122M    171G     0%    /usr/ports/distfiles
zroot/usr/src                172G    300M    171G     0%    /usr/src
zroot/var                    172G    159M    171G     0%    /var
/dev/md0                     124M     78K    114M     0%    /tmp
procfs                       4.0K    4.0K      0B   100%    /proc

I am no zfs expert but i think that the compression is transparent to userspace, ie: the loader doesn't have to actually decompress the kernel.
 
thug4life: did you turned on the compression on zfs root filesystem BEFORE installing everything, just after you did zpool create?

Did you ever recompiled the kernel?

I'm asking this because I can confirm the kernel DOES NOT LOAD if it's compressed.

And remember that turning compression ON after you got the files (=installed) you mean the files are still NOT compressed at all (the new setting apply only to new files, not to old ones...)

ZFS is transparent, but at booting its not the zfs code loading the kernel, but the tiny freebsd bootloader, which doesn't handle compressed kernel correctly.

I can confirm this, if you follow the tutorial it works, if I turn on compression... 'CANNOT LOAD KERNEL' on startup.
 
SirDice said:
Ah, right.. The key is probably in those last zfs set mountpount commands.
Not sure but you could try # zfs set mountpoint=legacy zroot/boot.
Click to expand...

If I do this zroot/boot filesystem may be 'seen' by boot, but after boot, /boot directory will not be mounted, because zfs do not mount fs with legacy mountpoint... so i will have no /boot -_-
 
I did a sync and rebuild a few hours ago.

Code: 
> zfs get -r mountpoint zroot
NAME                       PROPERTY    VALUE                      SOURCE
zroot                      mountpoint  legacy                     local
zroot/tmp                  mountpoint  /tmp                       local
zroot/usr                  mountpoint  /usr                       local
zroot/usr/ports            mountpoint  /usr/ports                 inherited from zroot/usr
zroot/usr/ports/distfiles  mountpoint  /usr/ports/distfiles       inherited from zroot/usr
zroot/usr/src              mountpoint  /usr/src                   inherited from zroot/usr
zroot/var                  mountpoint  /var                       local
 
thuglife said:
I did a sync and rebuild a few hours ago.

Code: 
> zfs get -r mountpoint zroot
NAME                       PROPERTY    VALUE                      SOURCE
zroot                      mountpoint  legacy                     local
zroot/tmp                  mountpoint  /tmp                       local
zroot/usr                  mountpoint  /usr                       local
zroot/usr/ports            mountpoint  /usr/ports                 inherited from zroot/usr
zroot/usr/ports/distfiles  mountpoint  /usr/ports/distfiles       inherited from zroot/usr
zroot/usr/src              mountpoint  /usr/src                   inherited from zroot/usr
zroot/var                  mountpoint  /var                       local
Click to expand...

Then I have to ask you how you built the system.

Could you please link to the guide you used or write here all the commands for installing your system?

Thank you for your help... with the official ZFSOnRoot guide its impossible to do it (CANNOT LOAD KERNEL error).
 
I had this EXACT same error recently when using mfsbsd and i thought i had done something wrong. I tried 2-3 times to get it to work and i ended up just doing it the way i KNEW would work, creating a ufs /boot and everything else ZFS.

If you track down what is causing this i'd like to know too.
 
wonslung how did you create the /boot?

Can you please link me all your settings (fstab, geom, zfs list, /boot/loader.conf, make.conf, etc...)?

I don't have any idea on how to mount a USF /boot on a ZFS root filesystem.
 
omero said:
Then I have to ask you how you built the system.

Could you please link to the guide you used or write here all the commands for installing your system?

Thank you for your help... with the official ZFSOnRoot guide its impossible to do it (CANNOT LOAD KERNEL error).
Click to expand...

Sorry to say but i followed the exact same guide :D
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

I use v14 with the latest gptzfsboot code, i am not sure if i had compression on or off before that.

For some reason i believe that compression has nothing to do with your inability to boot. I will have to test it in a vm.

...
But I noticed the kernel cannot be loaded from a ZFS filesystem with compression turned on.
Click to expand...

Can you post the exact error?
 
thuglife said:
Sorry to say but i followed the exact same guide :D
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

I use v14 with the latest gptzfsboot code, i am not sure if i had compression on or off before that.

For some reason i believe that compression has nothing to do with your inability to boot. I will have to test it in a vm.

Can you post the exact error?
Click to expand...

I use v13 with the 8-STABLE gptzfsboot code from january, from the last DVD ISO you can download.

I can *TOTALLY* assure you that if you follow the guide with that DVD iso, you boot fine and everything is ok.

If you instead turn on compression on zroot BEFORE starting to copy the boot stuff (so that the boot stuff, kernel, etc... gets compressed), you cannot boot and you get 'CANNOT LOAD KERNEL' at start.

My best guess is that 'something' in /boot doesn't work if compressed, maybe the kernel, maybe something else.

I invite you to do a small test with a VM, that's what I just did now AGAIN and I can confirm: following the guide but adding a

[CMD="zfs set compression=lzjb zroot"][/CMD]

just after having created the pool give you an UNBOOTABLE freebsd.

And also trying to separate /boot onto a different zfs filesystem does not work and does not boot.

So I don't have a solution yet: how to turn compression on zroot-wide for everything for default, and still retain a bootable system.
 
omero said:
wonslung how did you create the /boot?

Can you please link me all your settings (fstab, geom, zfs list, /boot/loader.conf, make.conf, etc...)?

I don't have any idea on how to mount a USF /boot on a ZFS root filesystem.
Click to expand...


what i did was this:
let's say you have a single hard drive and you want to make a ufs /boot a swap and the rest zfs

what you do is a normal sysinstall minimal install
create a 1024 MB /
whatever you need for swap
then create the rest in it's own giant partition (you may have to specify a mount point but you can use the m button to remove it in sysinstall, while making your slices so it's just a big empty slice)


you install the minimal system
then you will have a empty slice on d normally, like ad4s1d so you use that to make your zfs filesystem

Code: 
zpool create tank /dev/ad4s1d

then create some filesystems
Code: 
zfs create tank/root
zfs set mountpoint=none tank
zfs set mountpoint=/tank tank/root
zfs create tank/usr
zfs set mountpoint=/tank/usr tank/usr
zfs create tank/var
zfs set mountpoint=/tank/var tank/var
zfs create tank/tmp
zfs set mountpoint=/tank/tmp tank/tmp
zfs create tank/usr/local
zfs create tank/usr/ports
zfs create tank/usr/ports/distfiles

so on and so forth.

When you're done creating them, you should basically have all your filesystems set up with tank/root mounted on /tank and everything else mounted behind that.

Then you use cpio to copy the minimal installed base system to the new zfs filesystems like this:
Code: 
find -x / | cpio -pmd /tank

then make sure you add zfs_enable="YES" to /etc/rc.conf
then remove the zfs copy of boot because we won't need it, we're going to use the ufs filesystem
Code: 
rm -rf /tank/boot
now we'll make a mount point for the ufs filesystem and a symbolic link so it boots properly
Code: 
mkdir /tank/bootdir
cd /tank
ln -s bootdir/boot boot

then add the proper stuff to /boot/loader.conf
Code: 
echo 'zfs_load="YES"' >>  /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:tank/root"'  >>  /boot/loader.conf

now, if you edit the fstab file in the ZFS filesystem, you can tell it to mount the ufs filesystem on /bootdir so edit /tank/etc/fstab to this:

Code: 
/dev/ad4s1a  /bootdir        ufs     rw      1       1
(it used to look like this
Code: 
/dev/ad4s1a  /        ufs     rw      1       1
)


now if you've done everything right, you should be able to set your mountpoints correctly.
Code: 
# zfs set mountpoint=/tmp tank/tmp
zfs set mountpoint=/usr tank/usr
zfs set mountpoint=/var tank/var
and set the tank/root legacy
Code: 
cd /
zfs set mountpoint=legacy tank/root

heres one i did that way
Code: 
% zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
tank                      25.8G   427G    22K  none
tank/root                 21.4M   427G  21.4M  legacy
tank/tmp                  23.5K   427G  23.5K  /tmp
tank/usr                  25.6G   427G   385M  /usr
tank/usr/home               68K   427G    19K  /usr/home
tank/usr/home/wonslung      49K   427G    49K  /usr/home/wonslung
tank/usr/jails            22.8G   427G  1.57M  /usr/jails
tank/usr/jails/basejail    273M   427G   273M  /usr/jails/basejail
tank/usr/jails/forumjail    18K   427G    18K  /usr/jails/forumjail
tank/usr/jails/mysqljail    18K   427G    18K  /usr/jails/mysqljail
tank/usr/jails/seedjail   22.5G   427G  22.5G  /usr/jails/seedjail
tank/usr/local            1.58M   427G  1.58M  /usr/local
tank/usr/obj              1.76G   427G  1.76G  /usr/obj
tank/usr/ports             228M   427G   227M  /usr/ports
tank/usr/ports/distfiles   810K   427G   810K  /usr/ports/distfiles
tank/usr/src               510M   427G   510M  /usr/src
tank/var                  91.9M   427G  91.9M  /var
[wonslung@ks304611 /]% df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
tank/root                   427G     21M    427G     0%    /
devfs                       1.0K    1.0K      0B   100%    /dev
/dev/ad4s1a                 989M    862M     48M    95%    /bootdir
procfs                      4.0K    4.0K      0B   100%    /proc
tank/tmp                    427G      0B    427G     0%    /tmp
tank/usr                    427G    385M    427G     0%    /usr
tank/usr/home               427G      0B    427G     0%    /usr/home
tank/usr/home/wonslung      427G      0B    427G     0%    /usr/home/wonslung
tank/usr/jails              427G    1.5M    427G     0%    /usr/jails
tank/usr/jails/basejail     427G    273M    427G     0%    /usr/jails/basejail
tank/usr/jails/forumjail    427G      0B    427G     0%    /usr/jails/forumjail
tank/usr/jails/mysqljail    427G      0B    427G     0%    /usr/jails/mysqljail
tank/usr/jails/seedjail     450G     23G    427G     5%    /usr/jails/seedjail
tank/usr/local              427G    1.5M    427G     0%    /usr/local
tank/usr/obj                429G    1.8G    427G     0%    /usr/obj
tank/usr/ports              427G    227M    427G     0%    /usr/ports
tank/usr/ports/distfiles    427G    768K    427G     0%    /usr/ports/distfiles
tank/usr/src                428G    510M    427G     0%    /usr/src
tank/var                    427G     92M    427G     0%    /var
devfs                       1.0K    1.0K      0B   100%    /var/named/dev
/usr/jails/basejail         427G    273M    427G     0%    /usr/jails/seedjail/basejail
/usr/ports                  427G    227M    427G     0%    /usr/jails/seedjail/usr/ports
devfs                       1.0K    1.0K      0B   100%    /usr/jails/seedjail/dev
fdescfs                     1.0K    1.0K      0B   100%    /usr/jails/seedjail/dev/fd
procfs                      4.0K    4.0K      0B   100%    /usr/jails/seedjail/proc
 
THanks for the very detailed description.

I'll wonder and decide which solution is best. I would prefer to keep the system UFS-free, and use only ZFS.

I have to decide which one I like the most:

A) Using your solution
B) leaving root & boot uncompressed, while turning on compression on zfs filesysstem that i can create for EVERY root directory.
 
also, i'm not sure if it helps or not, but another option MIGHT be to get the opensolaris GRUB.

IT supports booting from compression=on systems (i'm not sure if it supports gzip)
 
wonslung said:
either option seems good to me.
what happens if you make boot it's own filesystem and put compression on /
Click to expand...

It happens it doesn't boot. Probably only the / gets mounted by gptzfsboot.
 
I've found out that this is because FreeBSD installs a ZFS loader in boot sector which by default loads:

"zroot:/boot/zfsloader".

If i press ESC i can change the default. Someone knows how to change the default automatically, so I can make a zroot/boot zfs with compression=off and all the rest with compression=lzjb?
 
omero said:
I've found out that this is because FreeBSD installs a ZFS loader in boot sector which by default loads:

"zroot:/boot/zfsloader".

If i press ESC i can change the default. Someone knows how to change the default automatically, so I can make a zroot/boot zfs with compression=off and all the rest with compression=lzjb?
Click to expand...



weird....it makes no sense to me why this should matter. I have no problem when i make a ufs /boot and everything else ZFS.
boots fine.
 
You must log in or register to reply here.
Back
Top