I get the impression from reading this thread that there may be some interesting stories to share on the subject of using or not using swap. Here's mine.
Once upon a time I was working at a place where we had several Hadoop clusters that ran on our own bare metal. Old school, I know. We suddenly started having problems with jobs failing or timing out. This was puzzling because the jobs that ran on the cluster hadn't changed, and we had actually upgraded the cluster software and hardware. We looked into the matter further, and discovered that the HDFS data nodes were dropping out and coming back due to network timeouts. This was also puzzling because our network infrastructure had not changed.
It turned out that some genius was running a Perl map/reduce job, and was relying on the fact that 32-bit Perl could use a maximum of 4GB of RAM. You can probably guess what happened at this point. The upgraded nodes had 64-bit Perl on them. The M/R job started taking insane amounts of memory. This memory pressure forced the HDFS daemons into swap because they were idle most of the time. Then the master node would send them a request which would time out because it took so long to restore the daemons' working set from spinning rust. Eventually the daemon would come back and say "I'm not dead yet!" to the master node. Everything was normal and snappy when the Perl job wasn't running.
We decided at that point that swap = bad, mmkay? and configured all Hadoop nodes without it forevermore. I wonder if that still applies in these days of fast NVMe "disks", though.
Once upon a time I was working at a place where we had several Hadoop clusters that ran on our own bare metal. Old school, I know. We suddenly started having problems with jobs failing or timing out. This was puzzling because the jobs that ran on the cluster hadn't changed, and we had actually upgraded the cluster software and hardware. We looked into the matter further, and discovered that the HDFS data nodes were dropping out and coming back due to network timeouts. This was also puzzling because our network infrastructure had not changed.
It turned out that some genius was running a Perl map/reduce job, and was relying on the fact that 32-bit Perl could use a maximum of 4GB of RAM. You can probably guess what happened at this point. The upgraded nodes had 64-bit Perl on them. The M/R job started taking insane amounts of memory. This memory pressure forced the HDFS daemons into swap because they were idle most of the time. Then the master node would send them a request which would time out because it took so long to restore the daemons' working set from spinning rust. Eventually the daemon would come back and say "I'm not dead yet!" to the master node. Everything was normal and snappy when the Perl job wasn't running.
We decided at that point that swap = bad, mmkay? and configured all Hadoop nodes without it forevermore. I wonder if that still applies in these days of fast NVMe "disks", though.