my zfs tips

graudeejs · #1 August 12th, 2009, 16:29

I've been using zfs for some time now on my desktop pc. And finally I decided to share some tips.

Before we start, you need to know that I use geli to encrypt my HDD's. I boot from usb flash. And what I tell may depend on that.

I don't intend to explain how to start using ZFS, I want to tell you how to use it and save some problems later.

Let's imagine we have fresh zpool named mzfs.

1) set zfs options for your pool that you think would be default in most cases
For example, I set:

Code:

zfs set copies=2 mzfs
zfs set compression=gzip mzfs

New zfs file systems will use zfs settings from parent file system (in this case zpool root), you can override them for other file systems

In addition I do

Code:

zfs set mountpoint=none mzfs

2) Don't install FreeBSD to root of pool.
create mzfs/root and install FreeBSD there.
[I will explain later]

3) group jails
To group my jails I create filesyste

Code:

zfs create mzfs/jails
zfs set mountpoint=/jails mzfs/jails

set other options as you need.

Child file systems will have parent fs settings, and will be auto mounted

4) create snapshot for 1 jail use for all of them
Now let's imagine you have build your system from sources and want to set up jails.
First thing you need to do is create 1 jails, that will be used for all other jails as starting point.

Code:

zfs create mzfs/jails/testbox
cd /usr/src
make buildworld
make installworld DESTDIR=/jails/testbox
cd etc
make distribution DESTDIR=/jails/testbox
cd /jails/testbox/etc
touch fstab

after this edit all files in /jails/testbox/etc that you nedd (rc.conf, resolv.conf)

once you have finished editing files, this is the point every jail will have common.
Create zfs snapshoot

Code:

zfs snapshot mzfs/jails/testbox@fresh

now, when you need fresh jail for you server simply run
zfs clone mzfs/jails/testbox@fresh mzfs/jails/server
this jail will be auto mounted

I also keep /jails/fstab.JAIL_NAME files (replace JAIL_NAME with jail name)
here's sample:

Code:

/usr/ports /jails/testbox/usr/ports nullfs noatime,rw 0 0
/usr/src /jails/testbox/usr/src nullfs noatime,ro 0 0

here's sample from my rc.conf

Code:

jail_testbox_hostname="testbox"
ifconfig_rl0_alias0="inet 192.168.128.99 netmask 0xffffff00"
jail_testbox_ip="192.168.128.99"
jail_testbox_rootdir="/jails/testbox"
jail_testbox_exec="/bin/sh /etc/rc"
jail_testbox_devfs_enable="YES"
jail_testbox_fstab="/jails/fstab.testbox"
jail_testbox_mount_enable="YES"

when I create new jail I simply copy this block and edit it.
and copy /jails/fstab.testbox to /jails/fstab.server

add new jail to jail list and start it.
creating new jails with zfs is very easy, and best of all it takes zero space.

5) Rebuild system from jail
Now If you want to have fresh FreeBSD, you can build it in jail
simply do what's written in tip 4

after you installed all apps from ports (or whatever you did)
stop jail and then edit loader.conf
change

Code:

vfs.root.mountfrom="zfs:mzfs/root"

to

Code:

vfs.root.mountfrom="zfs:mzfs/root2"

don't forget to edit your /jails/newfreebsd/etc/rc.conf and /jails/newfreebsd/etc/fstab, or copy them form your old freebsd

Code:

cp /etc/rc.conf /jails/newfreebsd/etc/rc.conf
cp /etc/fstab /jails/newfreebsd/etc/fstab

do same for other files as well, that you have modified.
and finally run

Code:

zfs rename zfs/jails/newfreebsd zfs/root2

next time you reboot you will boot in your newly created FreeBSD.
This way you will avoid downtime, and won't have to destroy zpool (pretty much no matter what)

after reboot you can destroy old system

Code:

zfs destroy a/root

6) you may not agree with this
On zfs I prefer to not separate /usr /usr/local etc from root. Basically I keep everything I need to run FreeBSD as Desktop under 1 Filsystem.
I separate /tmp /usr/src /usr/ports /usr/obj /var/db/portsnap becaue i use different zfs options on them (copies=1 for example)

NOTE: don't forget to install kernel on usb flash

[or the media you're using to boot from]

Fallowing these tips I installed fresh FreeBSD with KDE4.3, just to test KDE, and later switched back to my good old fvwm.

I attached tar with my config files, to help you understand what I was saying here.
Note, that my fstab is empty. It really is empty on my pc.

P.S.
If you have questions, post here, I'll answer

P.S.S.
Now think what would you do if you had your FreeBSD root on zpool root?

UPDATE:
7) compression=off
Compression sounds, good, but in reality it isn't all that good.
1.) you can't boot from compressed pool
2.) when i was copying data to compressed pool on flash I had huge lags (well compression was high, but still, lags ware HUGE, really HUGE). When I was copying same data, to same flash, but with compression turned off, lags were gone.

From now on, i will use compression only on very few parts of systems (ports tree, logs....)

UPDATE:
Remember that if you need to work with zfs in fixit mode (from fixit cd or dvd), you need to

Code:

# mkdir /boot/zfs
# ln -s /dist/boot/kernel /boot/kernel
# ln -s /dist/lib /lib
# kldload zfs

Also remember, that you probably won't be able to boot from exported zfs!

UPDATE:
If you want to make periodic zfs snapshots you may want to try sysutils/zfsnap

UPDATE:
in /etc/contab

Code:

@reboot root /usr/local/sbin/zfSnap -a 1w -r a/home a/root a/local

rbelk · #2 August 12th, 2009, 23:43

Thanks killasmurf86! I always look at the manuals when I do a ZFS install on our new Sun Servers. I have to configure ZFS once or twice a year so it is very rare for me to configure ZFS. But now that FreeBSD's ZFS has matured I'll be using it a lot more

.

**avilla@** · #3 August 28th, 2009, 15:19

Quote:

Originally Posted by killasmurf86

7) compression=off
Compression sounds, good, but in reality it isn't all that good.
1.) you can't boot from compressed pool

are you sure? i have compression=on on my root filesystem, and no problems at all

Quote:

2.) when i was copying data to compressed pool on flash I had huge lags (well compression was high, but still, lags ware HUGE, really HUGE). When was copying same data, to same flash, but with compression turned off, lags were gone.

this really depends on hardware and system usage. i'm using zfs on my core duo laptop (which is basically my desktop: music, videos, development, studying... and so on): it's faster while reading and writing data with iozone, and it takes the same time while copying media files to a compressed and uncompressed filesystem (with only 5-6% of different cpu load)

graudeejs · #4 August 28th, 2009, 17:11

Well, I also use encryption.... but perhaps you're right...
Anyway, for my good old Pentium 4, compression=off works better.
Thanks for sharing your experience.

**avilla@** · #5 September 6th, 2009, 01:45

thank you for sharing yours! reading you suggestions and thinking a bit about them, i was able to design my own way to upgrade my system

the only noticeable difference from your configuration is that i keep freebsd and the local stuff (ports, /var things, etc.) on different datasets:

Code:

# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank            44.1G  29.2G    18K  none
tank/home-albi  34.7G  29.2G  34.7G  /home/albi
tank/jail-a51    149K  29.2G   203M  /usr/jail/groomlake
tank/local      2.20G  29.2G  2.20G  /usr/local
tank/obj          18K  29.2G    18K  /usr/obj
tank/ports      3.65G  29.2G  3.65G  /usr/ports
tank/root        204M  29.2G   204M  legacy
tank/src         296M  29.2G   296M  /usr/src
tank/swap          3G  31.6G   654M  -
tank/var        73.0M  29.2G  73.0M  /var

jail-a51 is, of course, a clone of root (which has a /var directory... the tank/var dataset overwrites it because of databases and friends)
also, i moved /usr/compat to /usr/local/compat, so that i can get a fresh system at any time and, more interesting, i can replace /usr/local and /var when i like most (i work on kde's area 51 so i often need to test it, and i build it in the a51 jail, with a dataset for $jail/usr/local)

to sum up my commands (and to leave a reference for me

):

Code:

# cd /usr/src
# make update && make buildworld buildkernel
# zfs create -o mountpoint=/mnt tank/root-new
# make installkernel DESTDIR=/mnt
# mergemaster -piD /mnt (probably useless)
# make installworld DESTDIR=/mnt
# mergemaster -iD /mnt
# echo 'clear_tmp_enable="YES"' > /mnt/etc/rc.conf
# cp /etc/localtime /mnt/etc (and anything you'd like to have in your fresh jails)
# zfs snapshot tank/root-new@8.0-beta3-20090906
# cd /boot && cp loader.conf splash.bmp /mnt/boot
# cd /etc && cp $USEFUL_STUFF /mnt/etc
# cd /mnt
# mkdir cdrom
# ln -s usr/local/compat compat
# ln -s usr/home home
# mkdir boot/zfs && cp /boot/zfs/zpool.cache boot/zfs
# vi boot/loader.conf (set vfs.root.mountfrom="zfs:tank/root-new")
# zpool set bootfs=tank/root-new tank
# zfs set mountpoint=legacy tank/root-new

now, when i'll boot, i'll be in my new (not upgraded: new) freebsd, with my old /usr/(home/*|jail/*|local|obj|ports|src) and /var, which is becoming incredibly comfortable

hope this is someway useful

graudeejs · #6 September 6th, 2009, 08:35

Quote:

Originally Posted by xzhayon

jail-a51 is, of course, a clone of root (which has a /var directory... the tank/var dataset overwrites it because of databases and friends)
also, i moved /usr/compat to /usr/local/compat, so that i can get a fresh system at any time and, more interesting, i can replace /usr/local and /var when i like most (i work on kde's area 51 so i often need to test it, and i build it in the a51 jail, with a dataset for $jail/usr/local)

Just don't forget to change passwords in jails.... (to some random passwords at least)

**avilla@** · #7 September 8th, 2009, 01:20

Quote:

Originally Posted by killasmurf86

Just don't forget to change passwords in jails.... (to some random passwords at least)

well, of course, anyway my jails are not exposed to the net

coming back to tips, may i also suggest compression for swap on a zvol? have a look at this data:

Code:

# swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/zvol/tank/swap   3145728     416M     2.6G    14%

# zfs get compressratio,referenced tank/swap
NAME       PROPERTY       VALUE      SOURCE
tank/swap  compressratio  4.74x      -
tank/swap  referenced     86.9M      -

you can imagine how this makes swapping faster and faster!

here's how i configured my swap zvol:

Code:

# zfs get all tank/swap 
NAME       PROPERTY              VALUE                  SOURCE
tank/swap  type                  volume                 -     
tank/swap  creation              Mon Sep  7 20:29 2009  -     
tank/swap  used                  3G                     -     
tank/swap  available             30.0G                  -     
tank/swap  referenced            86.9M                  -
tank/swap  compressratio         4.74x                  -
tank/swap  reservation           none                   default
tank/swap  volsize               3G                     -
tank/swap  volblocksize          8K                     -
tank/swap  checksum              off                    inherited from tank
tank/swap  compression           on                     inherited from tank
tank/swap  readonly              off                    default
tank/swap  shareiscsi            off                    default
tank/swap  copies                1                      default
tank/swap  refreservation        3G                     local
tank/swap  primarycache          metadata               local
tank/swap  secondarycache        none                   local
tank/swap  usedbysnapshots       0                      -
tank/swap  usedbydataset         86.9M                  -
tank/swap  usedbychildren        0                      -
tank/swap  usedbyrefreservation  2.92G                  -
tank/swap  org.freebsd:swap      on                     local

graudeejs · #8 September 8th, 2009, 04:48

does swaping work well on freebsd already?

**avilla@** · #9 September 8th, 2009, 08:13

Quote:

Originally Posted by killasmurf86

does swaping work well on freebsd already?

probably, but swapping 1/5 of the data is obviously better

graudeejs · #10 September 8th, 2009, 08:51

I'm not using swap at all, for now (2.5G ram)
It's enough for me even when I build OOO

even with

Code:

vm.kmem_size="1100M"
vm.kmem_size_max="1100M"
vfs.zfs.arc_max="600M"

crsd · #11 September 18th, 2009, 08:32

Jail tips are very useful, thanks.

john_doe · #12 September 21st, 2009, 03:28

I think using changes from kmacy's branch with ZIO_USE_UMA enabled can improve I/O performance.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Solaris ZFS	ronaldprettyman	General	9	October 13th, 2009 15:24
ZFS v13	tanked	General	11	June 21st, 2009 04:27
Zfs	SuperMiguel	General	16	June 5th, 2009 19:29
zfs - to be or not to be	graudeejs	Off-Topic	47	May 26th, 2009 09:03
FreeBSD 7 and KDE 3 really laggy, need some tips.	Mikael_L	Installing & Upgrading	5	December 10th, 2008 12:26

The Following 8 Users Say Thank You to graudeejs For This Useful Post:
avilla@ (September 6th, 2009), crsd (September 18th, 2009), desnudopenguino (August 19th, 2009), ian-nai (February 18th, 2011), rbelk (August 12th, 2009), Sennaar (August 16th, 2010), xserg86 (August 17th, 2009), z0ran (September 6th, 2009)