ZFS swap and loader.conf

X

xy16644

I am in the process of setting up a new server with FreeBSD 9.2 with amd64. I am using ZFS on all the drives. I run ZFS on root and use geli to encrypt everything except /boot.

I enabled ZFS swap as follows: zfs create -V 8G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap but when I view top I can't see any swap! It just says:
Code: 
Swap:

I have rebooted the machine but this hasn't made any difference. On my old server I can see my swap size and how much of it is free. Any ideas?

Also, this is my first amd64 server with ZFS. On my i386 server (with 4 GB RAM) I had the following /boot/loader.conf options set:
Code: 
vfs.zfs.prefetch_disable="1"
vfs.root.mountfrom="zfs:zroot"
zfs_load="YES"
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="80M"
vfs.zfs.vdev.cache.size="5M"

On my new amd64 server (which has 16 GB RAM), I have only set the following so far:
Code: 
vfs.root.mountfrom="zfs:zroot"
zfs_load="YES"

I have read the ZFS Tuning Guide for FreeBSD but am still unsure if I should change any further ZFS settings for 16 GB RAM?

I'd appreciate any help on this! I have read loads of ZFS articles but there are so many opinions about this. I would like the ARC cache to use up to 10 or 12 GB of RAM as the servers services only use up to 2 GB RAM.
 
Sebulon said:
/etc/rc.conf
Code: 
zfs_enable="YES"

# zfs set secondarycache=off zroot/swap

Cheers!

/Sebulon
Click to expand...

When I try to run this command I get:
Code: 
cannot set property for 'zroot/swap': 'secondarycache' must be one of 'all | none | metadata'

So I changed it to: zfs set secondarycache=all zroot/swap but I still can't see my swap file when using top! What am I doing wrong?

zfs list:
Code: 
zroot/swap   2.06G  6.69G    16K  -

shows that I have a swap file but top still shows:
Code: 
Swap:

Is FreeBSD 9.2-STABLE different from -RELEASE when it comes to showing your ZFS swap size?

Also, I was thinking of setting the following in my /boot/loader.conf:
Code: 
vfs.zfs.arc_min="2048M"
vfs.zfs.arc_max="7680M"

vm.kmem_size_max="32G"
vm.kmem_size="24G"

This is for a 16 GB of memory system. How do these values look? Is it safe to set the vfs.zfs.arc_max to somehthing higher (like 12G)? Ideally I would like to use up to 12 GB of memory for the ARC and have about 2 GB of memory for services running and then have a further 2 GB of memory free for future growth.

Thanks all :)
 
Get rid of the kmem and arc lines. Let the OS auto-tune those as needed. Only limit the ARC if you have programs that need the RAM more than the OS does (like databases).

Don't use swap-on-ZFS. Just, don't. If you have to ask "why not?", really don't. :) Instead, create a separate swap partition on every disk, and let the OS use those for swap. Things will be faster and more stable than trying to make swap-on-ZFS work.
 
Thanks @phoenix!

In other words, I should just leave /boot/loader.conf alone when it comes to ZFS and let it do its own thing (autotuning)?

I've already built the server and used the (entire) disks for ZFS so I can't allocate a partition to swap. I was thinking of using the server with no swap at all, is this a bad idea? From what I have read ZFS will use up to 7/8 of RAM for ARC. So on a 16 GB system it will use 14 GB of RAM which leaves me just over 2 GB of RAM for services which is more than enough (I use about 1.4 GB maximum for services). Will no swap work? Or is this just a really bad idea?

PS: I've run swap on ZFS on my old server for almost two years now so I am surprised by you saying to not do this! :OOO
 
Last edited by a moderator:
Swap-on-ZFS works ... until you actually run out of RAM, and then everything goes wonky. For swapping out some memory to disk during normal operations, it works fine. Of course, running without swap during normal operations works fine too... until you actually run out of RAM and need swap. :)

ZFS needs RAM to track disk usage; system runs out of RAM needs to swap to disk; ZFS requests RAM to track writes to disk; system tries to page stuff out to disk to free up RAM for ZFS; and poof! Deadlock. There's been lots of fixes over the years to make this works better, but I wouldn't trust a live server with important data to it. :) There's even a thread on one of the mailing lists (Zaphod Beeblebrox started it I believe) about this very situation, with nasty results.

If there's no way to rebuild the server, then look into sticking an SSD in there. Use the majority of it for L2ARC (cache) and the rest for swap.

I find leaving loader.conf empty to start, and then tuning things after you see how things work, to be much better than trying to tune before you know how the system will work. If you run into a performance issue, then tune things. If you don't... then there's nothing to worry about.
 
Why thank you for the explanation @phoenix!

Rebuilding the server right now would be a real pain and take some time. I could redo the one ZFS pool and use part of the drive for swap but thats on a non SSD drive so I am not keen.

I like your idea of having an SSD drive for L2ARC. Its probably overkill for my needs but I just had a quick look online and can get a 60 GB Kingston SSD drive for about £37. How much space should I use on the SSD for swap? I have 16 GB RAM so should I create a 16 GB swap file? How do I create two partitions to be used for L2ARC and swap?

Oh and I want the swap encrypted :e I was going to follow this guide:

https://www.dan.me.uk/blog/2010/01/24/encrypted-swap-space/

So basically I want two partitions (using GPT):
  • 16 GB for swap space (or whatever the correct size is considering my RAM size)
  • 44 GB L2ARC

I'd like the L2ARC to be encrypted too with geli, is all this possible?

Thanks very much! :stud
 
Last edited by a moderator:
So in the end what I did was partition my two mirrored drives so that each drive has an 8 GB partition at the end of the drive used for swap. I then added the following to /etc/rc.conf:
Code: 
geli_swap_flags=”-e AES -l 128 -s 4096 -d”
and added the following to the /etc/fstab:
Code: 
# Device Mountpoint FStype Options Dump Pass#
/dev/da0p2.eli none swap sw 0 0
/dev/da1p2.eli none swap sw 0 0

So when I run top now I can see that I have 16 GB of swap space. My question is: If one of the drives fails in this mirror setup, will the system still carry on working? I.e.: not crash? The swap partitions aren't mirrored so I was wondering what would happen in the event of a disk failure. After much reading I decided against swap on ZFS!
 
You must log in or register to reply here.
Back
Top