Solved Laundry Memory, Jails, and Thrashing

Good Morning All,
I have been experiencing some mysterious thrashes on one of my servers, and I have finally gathered enough data about the problem to possibly ask for help (it was very difficult to get data since once it started thrashing there was nothing to be done except reboot...).

The machine has 32 GB of RAM, and an 8 GB swap partition with a latest generation Xeon 1270. Its tasked with running 5 instances of selenium+firefox in jails in an automated fashion. Initially, I had seen problems/thrashing from Firefox grabbing too much RAM (at one point I saw a firefox process with 14.9 GB of reserved memory... easily the largest memory I have ever seen a single process grab). I deduced this was due to the new Firefox quantum versions being more RAM heavy to improve speed, and the fact you are not really supposed to run more than one instance of firefox at a time (the firefox launcher will prevent you from launching a second instance if it can see another, in this case it did not see the other instances because they were in seperate jails). I attempted to solve the problem with rctl() rules to limit each jail to 4 GB of RAM. This was successful in eliminating the massive memory grabs from Firefox, but did not eliminate the thrashes.

I was able to watch this play out in top() and found that with the rctl limit in place, Firefoxes never exceeded 4 GB, but when they attempted to grab memory, the "Laundry" memory would spike dramatically. I watched it climb to 25 GB with 50% of the swap used before the machine froze up in a thrash state.

I suspect rctl may not be the best way to handle the memory limits for the jails, or perhaps there is a way to tune the system to prioritize freeing "Laundry", although I am not really clear as to exactly what circumstances cause for FreeBSD to mark a page as "Laundry" instead of "Inactive".

Unfortunately, this is extremely difficult to reproduce as it happens at the whim of Firefox going wild with the malloc calls. As always, I am most thankful for your expertise.
 
Did you also restrict ZFS arc cache memory usage? Because this tends to grab all memory it can, too.
 
I think I may have gotten to the bottom of the problem. It looks like browsers aren't just memory hogs, they're disk i/o hogs as well! It seems as the firefox instances wantonly grab disk, they drive up the "Laundry" stack. So far its been 24 hours without "Laundry" exceeding 2 GB after limiting jail disk i/o to a reasonable amount. Will mark as solved if it goes for the whole weekend without any signs of thrash.
 
Can you try a different browser?

Unless firefox is doing something heinously complex (huge numerical simulation implemented in Javascript?), there is absolutely no logical reason for it to use multiple gigabytes. The screen has only a few million pixels, the HTML file it's rendering is probably at most a MB in size, and even with rendering data structures that's only good for a few dozen MB. I think Firefox has been coded to hog memory madly (designed for a single-user single-application environment), and probably also has a few memory "leak" bugs.
 
I think Firefox has been coded to hog memory madly (designed for a single-user single-application environment)
This sums it up. I think its programmed to see a main memory of 32 GB and says something along the lines of "cool I can malloc 16 GB". Unfortunately I cannot us a different browser, but I have succeeded in solving this problem by putting a restriction on both the virtual memory and the reserved memory per jail, along with the maximum i/o. This seems to have prevented FF from "seeing" that the system has 32 GB.

Interestingly, the performance of Firefox when restricted to 2 GB is about the same as when I run it on my desktop (16 GB) is about the same... Perhaps I should jail my desktop browser in a similar manner since it seems like its grabbing RAM with little performance benefit.
 
Interestingly, the performance of Firefox when restricted to 2 GB is about the same as when I run it on my desktop (16 GB) is about the same... Perhaps I should jail my desktop browser in a similar manner since it seems like its grabbing RAM with little performance benefit.

It's just a wild guess, but maybe Firefox is storing Caches in RAM. If so, it would make sense that you don't really notice any difference if you have a fast internet connection.
 
Back
Top