Should I mirror swap? Or no swap?

B

Bren

I recently moved all our production files servers from FreeNAS to FreeBSD 12.0-RELEASE. One thing that I have never been able to determine is whether or not to mirror swap. None of my research here or elsewhere concluded with a solid answer. Well, based on some recent experiences, I'm even more confused than ever.

I know the installer warns that if you mirror swap it disables crash dumps or something. What is the significance of this? What other issues might come up if I mirror swap? I haven't been able to find a definitive resource on this.

I've had issues with swap on two systems recently. One system has zroot on six spindles (no dedicated OS drives). I chose not to mirror swap so the installer created one swap partition per drive. I lost one of the drives the other day and the system became unstable and had to be rebooted. The drive was "stuck." "swapoff -a" wasn't working. Processes were dying with the following error:

Code: 
Dec 17 05:30:19 kern.crit backups01 kernel: swap_pager: I/O error - pagein failed; blkno 2622072,size 4096, error 6
Dec 17 05:30:19 kern.crit backups01 kernel: vm_fault: pager read error, pid 26086 (inetd)

There was no way around this that I could find except a reboot. Not good. Luckily this was just a backup system, but my production system is set up similarly so now I'm nervous.

Another system has dedicated 16GB SATA DOMs for boot and eight data drives. I believe what I did there was to create NO swap on the OS drives then later I set up geli + gmirror on the data drives. One day I pulled a drive in order to test something. After I put the drive back and onlined it ZFS was fine, but it messed up swap just like what happened with the system above so I had to reboot the box. This was very surprising to me.

The swap setup should have looked something like this:

Code: 
# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli   2097152     6716  2090436     0%
/dev/mirror/swap1.eli   2097152     6716  2090436     0%
/dev/mirror/swap2.eli   2097152     6716  2090436     0%
/dev/mirror/swap3.eli   2097152     6716  2090436     0%
Total             8388608 26864 8361744 0%

But it looked something like this:

Code: 
# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli   2097152     6716  2090436     0%
/dev/mirror/swap1.eli   2097152     6716  2090436     0%
/dev/da0p1
/dev/mirror/swap3.eli   2097152     6716  2090436     0%
Total             8388608 26864 8361744 0%

da0 being the drive I pulled and replaced. It looks like it broke the mirror and then grabbed the raw device for some reason. I was getting the same "pagein failed" errors above and had to reboot like I said.

So now I'm not sure what to do. On my Linux systems, I always like to keep a little swap and then I tune it so swapping is very rare (and everything has plenty of memory of course). I could probably run without swap and so far I have on the second system without any problems, but it's just a backup system.

What are your real world experiences with swap? My primary goal is to be able to lose a drive and not have the system become unstable, needing a reboot, of course. Mirroring swap doesn't seem to work as expected either! I'm inclined to remove swap at this point from all of my systems. Thoughts?
 
Swap just adds some extra relatively slow storage to the total pool of virtual address memory space.

If you install enough memory, you don't need swap. If you don't install enough memory your system will be unable to launch new processes or grow old ones (Google: OOM).

Since "enough memory" may be a nebulous concept, and disk space is cheap, most people add some swap. It's simple insurance.

I never build systems that don't have redundancy in the disk space. Having one dead disk take down a system is unacceptable. The more experienced I get, the more redundancy I want.

I generally keep my root file systems physically separate from the data. It gives me the control I want to upgrade or completely replace the operating system.

Where I put the swap depends on the size and use of the system. It might be in the space reserved for the root, or it might be in the ZFS "tank".

If I had eight data drives I would configure them in RAIDZ2 format and create a zvol for swap. From that, you would get premium performance from the striping plus premium redundancy (both quite superior to using mirrors).

There's not enough information for me to comment on what happened to your swap space mirror.
 
Yes that's why I always like adding a little swap space. The drives are all in RAID-Z2. I didn't consider doing a zvol for swap. I will look into that. Thanks for your input.
 
Hrmm...did a little research on using zvols for swap. Looks like doing so comes with some serious caveats. A FreeBSD dev cautioned against it in this bug report:

Without some serious work, swap on ZFS will be error prone as ZFS can require memory to complete its IO pipeline and swap data to its vdevs so if memory is really short it can easily cause a deadlock.

So while light use of swap on ZFS can work, heavy use of it will almost certainly fail.

Given this I would never recommend ZFS backed swap.
Click to expand...

There are some tweaks one can do but this only seemed to complicate things further.

In a VM I set up mirrored encrypted swap and removed and added the drives several times. I didn't have a problem with swap whatsoever (like I did with the one system I mentioned in the OP). I think I will move forward with this setup on all our systems as it's supported in the installer and seems to be one of the safer/simpler swap configurations.
 
The FreeBSD bug report is closed, so I had a good look around the Internet, and found the same issue raised, and unresolved (e.g. in Proxmox).

It seems that swap on zvol is still compromised by deadlock on memory allocation issues.

Well spotted.
 
My home server, a HPE ProLiant ML110 Gen10 with 2x1TB SATA running a ZFS Mirror, runs with "mirrored" swap. However, this happens with a swap file on zroot.

I'd say mirror swap if you use a swap partition, or put the swap file on a mirrored partition if you do that (like me). When a disk fails, your system doesn't. Without mirroring swap, if the disk with the swap fails, and the swap is in use, you're inviting serious trouble.

The only time you shouldn't mirror swap is if your root disk isn't mirrored.
 
I just rolled out mirrored, encrypted swap to all our servers. We'll see what happens when drives start to fail.
 
You must log in or register to reply here.
Back
Top