1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

my zfs tips

Discussion in 'Howtos and FAQs (Moderated)' started by graudeejs, Aug 12, 2009.

  1. graudeejs

    graudeejs Well-Known Member

    Thanks Received:
    I've been using zfs for some time now on my desktop pc. And finally I decided to share some tips.

    Before we start, you need to know that I use geli to encrypt my HDD's. I boot from usb flash. And what I tell may depend on that.

    I don't intend to explain how to start using ZFS, I want to tell you how to use it and save some problems later.

    Let's imagine we have fresh zpool named mzfs.

    1) set zfs options for your pool that you think would be default in most cases

    For example, I set:
    zfs set copies=2 mzfs
    zfs set compression=gzip mzfs

    New zfs file systems will use zfs settings from parent file system (in this case zpool root), you can override them for other file systems

    In addition I do
    zfs set mountpoint=none mzfs

    2) Don't install FreeBSD to root of pool.
    create mzfs/root and install FreeBSD there.
    [I will explain later]

    3) group jails
    To group my jails I create filesyste
    zfs create mzfs/jails
    zfs set mountpoint=/jails mzfs/jails

    set other options as you need.

    Child file systems will have parent fs settings, and will be auto mounted

    4) create snapshot for 1 jail use for all of them
    Now let's imagine you have build your system from sources and want to set up jails.
    First thing you need to do is create 1 jails, that will be used for all other jails as starting point.
    zfs create mzfs/jails/testbox
    cd /usr/src
    make buildworld
    make installworld DESTDIR=/jails/testbox
    cd etc
    make distribution DESTDIR=/jails/testbox
    cd /jails/testbox/etc
    touch fstab

    after this edit all files in /jails/testbox/etc that you nedd (rc.conf, resolv.conf)

    once you have finished editing files, this is the point every jail will have common.
    Create zfs snapshoot
    zfs snapshot mzfs/jails/testbox@fresh

    now, when you need fresh jail for you server simply run
    zfs clone mzfs/jails/testbox@fresh mzfs/jails/server
    this jail will be auto mounted

    I also keep /jails/fstab.JAIL_NAME files (replace JAIL_NAME with jail name)
    here's sample:
    /usr/ports /jails/testbox/usr/ports nullfs noatime,rw 0 0
    /usr/src /jails/testbox/usr/src nullfs noatime,ro 0 0

    here's sample from my rc.conf
    ifconfig_rl0_alias0="inet netmask 0xffffff00"
    jail_testbox_exec="/bin/sh /etc/rc"

    when I create new jail I simply copy this block and edit it.
    and copy /jails/fstab.testbox to /jails/fstab.server

    add new jail to jail list and start it.
    creating new jails with zfs is very easy, and best of all it takes zero space.

    5) Rebuild system from jail
    Now If you want to have fresh FreeBSD, you can build it in jail
    simply do what's written in tip 4

    after you installed all apps from ports (or whatever you did)
    stop jail and then edit loader.conf


    don't forget to edit your /jails/newfreebsd/etc/rc.conf and /jails/newfreebsd/etc/fstab, or copy them form your old freebsd
    cp /etc/rc.conf /jails/newfreebsd/etc/rc.conf
    cp /etc/fstab /jails/newfreebsd/etc/fstab

    do same for other files as well, that you have modified.
    and finally run
    zfs rename zfs/jails/newfreebsd zfs/root2

    next time you reboot you will boot in your newly created FreeBSD.
    This way you will avoid downtime, and won't have to destroy zpool (pretty much no matter what)

    after reboot you can destroy old system
    zfs destroy a/root

    6) you may not agree with this
    On zfs I prefer to not separate /usr /usr/local etc from root. Basically I keep everything I need to run FreeBSD as Desktop under 1 Filsystem.
    I separate /tmp /usr/src /usr/ports /usr/obj /var/db/portsnap becaue i use different zfs options on them (copies=1 for example)

    NOTE: don't forget to install kernel on usb flash ;) [or the media you're using to boot from]

    Fallowing these tips I installed fresh FreeBSD with KDE4.3, just to test KDE, and later switched back to my good old fvwm.

    I attached tar with my config files, to help you understand what I was saying here.
    Note, that my fstab is empty. It really is empty on my pc.

    If you have questions, post here, I'll answer

    Now think what would you do if you had your FreeBSD root on zpool root?

    7) compression=off
    Compression sounds, good, but in reality it isn't all that good.
    1.) you can't boot from compressed pool
    2.) when i was copying data to compressed pool on flash I had huge lags (well compression was high, but still, lags ware HUGE, really HUGE). When I was copying same data, to same flash, but with compression turned off, lags were gone.

    From now on, i will use compression only on very few parts of systems (ports tree, logs....)

    Remember that if you need to work with zfs in fixit mode (from fixit cd or dvd), you need to
    # mkdir /boot/zfs
    # ln -s /dist/boot/kernel /boot/kernel
    # ln -s /dist/lib /lib
    # kldload zfs

    Also remember, that you probably won't be able to boot from exported zfs!

    If you want to make periodic zfs snapshots you may want to try sysutils/zfsnap

    in /etc/contab
    @reboot root /usr/local/sbin/zfSnap -a 1w -r a/home a/root a/local

    Attached Files:

    ian-nai, Sennaar, crsd and 5 others thank for this.
  2. rbelk

    rbelk Member

    Thanks Received:
    Thanks killasmurf86! I always look at the manuals when I do a ZFS install on our new Sun Servers. I have to configure ZFS once or twice a year so it is very rare for me to configure ZFS. But now that FreeBSD's ZFS has matured I'll be using it a lot more :p.
  3. avilla@

    avilla@ New Member Developer

    Thanks Received:
    are you sure? i have compression=on on my root filesystem, and no problems at all

    this really depends on hardware and system usage. i'm using zfs on my core duo laptop (which is basically my desktop: music, videos, development, studying... and so on): it's faster while reading and writing data with iozone, and it takes the same time while copying media files to a compressed and uncompressed filesystem (with only 5-6% of different cpu load)
  4. graudeejs

    graudeejs Well-Known Member

    Thanks Received:
    Well, I also use encryption.... but perhaps you're right...
    Anyway, for my good old Pentium 4, compression=off works better.
    Thanks for sharing your experience.
  5. avilla@

    avilla@ New Member Developer

    Thanks Received:
    thank you for sharing yours! reading you suggestions and thinking a bit about them, i was able to design my own way to upgrade my system

    the only noticeable difference from your configuration is that i keep freebsd and the local stuff (ports, /var things, etc.) on different datasets:

    # zfs list
    tank            44.1G  29.2G    18K  none
    tank/home-albi  34.7G  29.2G  34.7G  /home/albi
    tank/jail-a51    149K  29.2G   203M  /usr/jail/groomlake
    tank/local      2.20G  29.2G  2.20G  /usr/local
    tank/obj          18K  29.2G    18K  /usr/obj
    tank/ports      3.65G  29.2G  3.65G  /usr/ports
    tank/root        204M  29.2G   204M  legacy
    tank/src         296M  29.2G   296M  /usr/src
    tank/swap          3G  31.6G   654M  -
    tank/var        73.0M  29.2G  73.0M  /var

    jail-a51 is, of course, a clone of root (which has a /var directory... the tank/var dataset overwrites it because of databases and friends)
    also, i moved /usr/compat to /usr/local/compat, so that i can get a fresh system at any time and, more interesting, i can replace /usr/local and /var when i like most (i work on kde's area 51 so i often need to test it, and i build it in the a51 jail, with a dataset for $jail/usr/local)

    to sum up my commands (and to leave a reference for me :)):

    # cd /usr/src
    # make update && make buildworld buildkernel
    # zfs create -o mountpoint=/mnt tank/root-new
    # make installkernel DESTDIR=/mnt
    # mergemaster -piD /mnt (probably useless)
    # make installworld DESTDIR=/mnt
    # mergemaster -iD /mnt
    # echo 'clear_tmp_enable="YES"' > /mnt/etc/rc.conf
    # cp /etc/localtime /mnt/etc (and anything you'd like to have in your fresh jails)
    # zfs snapshot tank/root-new@8.0-beta3-20090906
    # cd /boot && cp loader.conf splash.bmp /mnt/boot
    # cd /etc && cp $USEFUL_STUFF /mnt/etc
    # cd /mnt
    # mkdir cdrom
    # ln -s usr/local/compat compat
    # ln -s usr/home home
    # mkdir boot/zfs && cp /boot/zfs/zpool.cache boot/zfs
    # vi boot/loader.conf (set vfs.root.mountfrom="zfs:tank/root-new")
    # zpool set bootfs=tank/root-new tank
    # zfs set mountpoint=legacy tank/root-new

    now, when i'll boot, i'll be in my new (not upgraded: new) freebsd, with my old /usr/(home/*|jail/*|local|obj|ports|src) and /var, which is becoming incredibly comfortable

    hope this is someway useful ;)
  6. graudeejs

    graudeejs Well-Known Member

    Thanks Received:
    Just don't forget to change passwords in jails.... (to some random passwords at least)
  7. avilla@

    avilla@ New Member Developer

    Thanks Received:
    well, of course, anyway my jails are not exposed to the net

    coming back to tips, may i also suggest compression for swap on a zvol? have a look at this data:

    # swapinfo -h
    Device          1K-blocks     Used    Avail Capacity
    /dev/zvol/tank/swap   3145728     416M     2.6G    14%
    # zfs get compressratio,referenced tank/swap
    NAME       PROPERTY       VALUE      SOURCE
    tank/swap  compressratio  4.74x      -
    tank/swap  referenced     86.9M      -

    you can imagine how this makes swapping faster and faster!

    here's how i configured my swap zvol:

    # zfs get all tank/swap 
    NAME       PROPERTY              VALUE                  SOURCE
    tank/swap  type                  volume                 -     
    tank/swap  creation              Mon Sep  7 20:29 2009  -     
    tank/swap  used                  3G                     -     
    tank/swap  available             30.0G                  -     
    tank/swap  referenced            86.9M                  -
    tank/swap  compressratio         4.74x                  -
    tank/swap  reservation           none                   default
    tank/swap  volsize               3G                     -
    tank/swap  volblocksize          8K                     -
    tank/swap  checksum              off                    inherited from tank
    tank/swap  compression           on                     inherited from tank
    tank/swap  readonly              off                    default
    tank/swap  shareiscsi            off                    default
    tank/swap  copies                1                      default
    tank/swap  refreservation        3G                     local
    tank/swap  primarycache          metadata               local
    tank/swap  secondarycache        none                   local
    tank/swap  usedbysnapshots       0                      -
    tank/swap  usedbydataset         86.9M                  -
    tank/swap  usedbychildren        0                      -
    tank/swap  usedbyrefreservation  2.92G                  -
    tank/swap  org.freebsd:swap      on                     local
  8. graudeejs

    graudeejs Well-Known Member

    Thanks Received:
    does swaping work well on freebsd already?
  9. avilla@

    avilla@ New Member Developer

    Thanks Received:
    probably, but swapping 1/5 of the data is obviously better
  10. graudeejs

    graudeejs Well-Known Member

    Thanks Received:
    I'm not using swap at all, for now (2.5G ram)
    It's enough for me even when I build OOO

    even with
  11. crsd

    crsd New Member

    Thanks Received:
    Jail tips are very useful, thanks.
  12. john_doe

    john_doe Guest

    I think using changes from kmacy's branch with ZIO_USE_UMA enabled can improve I/O performance.