swap - memory

Mage

Member

Reaction score: 16
Messages: 98

Could you try to enable compression on the zvol, or swap on a physical device?

And you may need to check the block size of the ZVOL, swapping is done using 4k blocks. I can't look up the settings on my machine as I am not at home.
The block size was 8k. I never touched it. I guess it’s the default. However, I used to have a few recordsize=8k (and even 4k) file systems due to wide-spread advice. It was a horrible mistake. Later, when I tested it, I could not find a single scenario where a non-128k recordsize was better than the default 128k. I tested it with PostgreSQL, Exim, and various logfiles. I tested it with and without compression.

As for putting the swap on a physical device, I have no interest in it since almost all my production servers are running "in the cloud". Even if I wanted to test it (I don’t), there isn’t a single computer on Earth which fulfills all of these: has a physical drive, has FreeBSD on it, I have access to it, it’s not a production machine.

By the way, many years ago, the main reason why I migrated to FreeBSD was ZFS. The main reason I migrated to ZFS was the checksumming. I would never put swap on a physical drive under FreeBSD. It makes zero sense to have checksums on the disk if the swap is not checksummed. I would compare it to putting ZFS on a physical raid. No, it’s worse. ZFS on a physical raid at least would notice if there was an error.
 

Mage

Member

Reaction score: 16
Messages: 98

I have even worse experience with swap and VMs.
Somehow I could understand Crivens’ desire to make me test it on a physical disk. I like physical servers. Pricing is better. Performance is better. I even like to know that there is somewhere a machine that’s mine. But there is a point when the flexibility worth more than the performance advantage.

The point is, if the production is running in a VM, the best place to run tests is a VM.

After a day or two all VM memory is moved to swap. My server has 64GB and 32 GB swap and usually, the layout looks like 28GB of used swap and everything is ARC. Not only VMs are slow, but I have OOM events as well. Not sure what to do.
Turning off the swap?
 

abishai

Aspiring Daemon

Reaction score: 151
Messages: 675

Swap is absolutely mandatory in case of ZFS filesystem if ARC size is not limited: in case of memory usage spike from userland apps, ZFS subsystem has good chances to die before it shrinks ARC.
 

Mage

Member

Reaction score: 16
Messages: 98

Swap is absolutely mandatory in case of ZFS filesystem if ARC size is not limited: in case of memory usage spike from userland apps, ZFS subsystem has good chances to die before it shrinks ARC.
I understand what you say. The only question is, why my testing showed me the opposite of this.

It’s a bit similar to the recordsize. I took common advice about ZFS. Later, when I tested it, I experienced the opposite. Okay, this is not limited to ZFS but the life on this planet.

My testing methods were clear. Although there was a bit self-irony in the "super scientific" labeling, two things I often/all the time run on my servers are Ruby and xz. As my production servers are running in a VM, I ran my tests in a VM. I did it multiple times.

Maybe there are cases when you are right. However, I know that at least there are cases when the opposite is true. I don’t get it why unbacked opinions spread on technical forums.

I ran through the five pages of this thread before I posted here. I didn’t read all posts but at least I looked at them. It’s like 70% of the posts are telling that if one runs FreeBSD without a swap partition, the Universe will collapse into itself, or at least the dimensional gates will shatter and release unholy creatures to destroy life on Earth.

And maybe 30% of the posts say they turned off the swap, and it’s either fine or better.

I’m busy too like many people but it’s still possible to spend an hour on testing before I post something that might affect others’ decisions. Besides, I have seen my production servers swapping while the ARC was dozens of GBs.
 

Mage

Member

Reaction score: 16
Messages: 98

Update: after I removed the swap and the limit of the ARC, one of my servers were running near 100% CPU usage all the time. The other servers also exhibited increased CPU usage.

At first attempt, I didn’t turn on the swap but I put back the limit on the ARC. Right after I did it, the average CPU usage went from 98% to 40%. It has been running like that for more than six hours.

Maybe it’s not the swap that has an issue but the ARC. It doesn’t mean that swap is needed. It’s interesting that the impact on the servers varies. It was the worst on the server that has the most memory. These servers run the same software and are under similar workloads. They are running web and PostgreSQL. They have tons of RAM. Most of the RAM is used by the OS for caching either as ARC or buffer cache.
 
Top