Problems with swap on ZFS ZVOL?

nsayer

nsayer

I've recently set up a VPS at rootbsd. I set it up with GPT+ZFS almost the same way as my old physical machine (the difference is that I'm not using a mirror for the tank, since the underlying storage has redundancy, and the virtual disk is much smaller than the physical disk I had before): the only things on the disk are the freebsd-boot partition and the tank. Swap is provided on a zvol.

The system only has 768 MB of RAM, but this page has some tuning that suggests that it ought to work.

Under load, the system winds up using a couple dozen MB of swap, but then all of a sudden it locked up with repeated messages:

Code: 
swap_pager: indefinite wait buffer: bufobj:

All I get from teh g00g on this is a thread from September of 2007 on freebsd-current.

Is there still problems swapping to a zvol? What can I do to make it more reliable?
 
I checked out the page you referenced. Turning checksums off is a good idea. I'll do that. I also read elsewhere that creating the zvol with a 4k block size was a good idea.

The example also creates a traditional swap partition. I didn't do that, and now it would be a day of work to repartition the drive. But if you're not usig that swap partition, then your results should still be valid.

But this is a very RAM constrained (virtual) machine. I wonder if that's the difference.
 
danbi said:
Some people say enabling compression on the swap volume helps as well.
Click to expand...

That's counter-intuitive. Do you have any references that explain why?
 
On one hand you need to read less data... on slow disks and fast cpu, this could be beneficial (perfect for floppy disks :D).
On other hand you need to decompress data, which means, more cpu cycles, which means bigger latency.
 
No, this is still going on. I just caught my VPS stuck with this same problem.

The provider gave us all a RAM upgrade for free not too long ago. It's up to 1.3 GB of RAM, but that still isn't a lot for a ZFS equipped server.

I have a single zvol for swap, but things like building a JVM from the ports will easily wind up getting it stuck this way.
 
You are using ZFS on root on a very low memory system. (less significant in this case)
You are using ZFS on a virtualized vdev (more significant in this case)

Backup and reinstall using UFS or UFS+J.

If you want to implement a ZFS on root then get a dedicated server.
 
So this is continuing to happen on FreeBSD 9.2. I can't buildworld 9.3 with the system running multi-user because with swap on a zvol it hangs, and without swap it runs out of RAM.

Is swapping on a zvol ever going to work properly?
 
I'm sure there are a few posts about problems with swap on ZFS on here. If at all possible, just don't do it. Create a completely separate partition during install and just use that like any other normal install.

The machine uses swap when it's low on RAM, but all writes to ZFS are cached in RAM for 5-30 seconds before being flushed to disk as a single transaction, so you're trying to use RAM in order to write data to disk, when you're trying to use disk because you're out of RAM...

I have no doubt there are possibly things that could be improved for swap on ZFS, but in the end it just is not appropriate to use something higher level like ZFS, that requires CPU and RAM resources, for swap. The system should just have direct access to a swap partition on the disk.
 
Swap on ZFS has been shown to be a bad idea, but exponentially so when you have memory constraints. Minimum recommended RAM is 4 GB for ZFS, and although ZFS can work with less than 4 GB, throwing swap into this is going to give you problems.
 
I did go through the flags and noticed that primarycache and secondarycache were not set to disabled. I've changed that and will try that out.

Unfortunately, I have read all of the advice not to swap onto a zvol, but at this point reconfiguring this (virtual) machine would be too painful, so getting it to work minimally acceptably would be best.
 
Turns out it still deadlocks... it just doesn't deadlock with the "indefinite wait buffer" message on the console.

I tried setting vfs.zfs.arc_max to 128M 128 MB (the system has 1.5G 1.5 GB of RAM). Are there any other settings I could try?
 
It doesn't work properly in the versions you mentioned even if you have a lot of memory. It will deadlock when it starts swapping, although not necessarily immediately. This may be fixed in the long run but for now a swap partition (or swap file on UFS) is the only reliable solution as far as I know.
 
You must log in or register to reply here.
Back
Top