swap - memory

ronaldlees

Aspiring Daemon

Thanks: 297
Messages: 708

#51
For the moment I'm trying to limit a user *total* memory use. In such a way that, even if i would set "swap off", a general user "foo" could not put the system out of memory and crash all.
+1 - Let us know how your experiments work out.

I think that FreeBSD is very conservative, even for the "server" setting that is currently "stiffly" soft-wired. They want "hot" processes to get the juice, and to make sure that filesystem stuff is never reloaded, and the paging to other queues is kept down for IO throughput. So - it's a server with a network bias.

But, would be nice to have vm.server_bias=0 or some such (bad name) - switch for us GUI guys.
 

ronaldlees

Aspiring Daemon

Thanks: 297
Messages: 708

#52
1] If you read "man rctl", and then do some experiments, you will see it is not self evident how to reach this : "not swap any X based processes". BTW, IMHO, the easiest way is to limit a specific user.
Untill now I have not found a way.

2] Yes, in my opinion Firefox memory management is inapproriate, how do I fill "bug?" report to the person who manages the FreeBSD port ? (sorry i'm new)
Look at the vm.overcommit sysctl value. I think if you want RLIMIT_SWAP to work, then you need bit one of vm.overcommit set, but by default it is set to zero. Of course you could still lose data if your user app crashes. :)

See:
https://people.freebsd.org/~kib/overcommit/

There may be other side effects of this setting, so you'll have to be wary. I see that literally nobody is doing this (including me), based on web searches, so you'd be on your own relative to the risks, etc.
 

Nicola Mingotti

Well-Known Member

Thanks: 123
Messages: 312

#53
ronaldlees, thank you ! I will read carefully later.

I just finished a couple of experiments which I would really like some of you
to confirm because I found interesting news (news at least for me) but also scary news.

NOTE: there experiments where made WITH swap-on. WITHOUT X, only 3 terminals
in a small virtual machine running FreeBSD-11.1.

1] The previous version of "memEater.c" I posted is completely harmless,
I saw that I can (m)allocate more then double of the phisical memory I have.
The system here is smart, he sees that I allocate memory and I am not using it
so it lets me do whatever I want.

1.1] The memory "memEater.c" uses does not appear in "top", but you can see it in rctl -hu user:foo under the name "swapuse" ==> temporary conclusion, it is useless to limit "swapuse" because it can grow out of measure, well beyond system capabilities.

2] I wrote a second version of "memEater.c", called "memEater2.c" (it is attached ) which not only allocates big chunks of memory but also [great stuff;)] fill it with 'a' ! This makes a lot of difference !

2.1] Using "memEater2.c" you will see memory usage growing in top

2.2] Using "memEater2.c" you will see "memoryuse" to grow in rctl -hu user:foo output under the name "memoryuse"

2.3] => "memoryuse" it is probably only thing which has some sense to constrain.

3] Using "memEater2.c" at the point of eating more memory than it is phisically available in Ram + Swap an ugly thing happened. "memEater2.c" kept trying to eat stuff, it was not terminated by the system and I lost control of the machine. There appeared an error message in the "root" console saying something about memory, but the teminal was lost.

4] At this point I can confirm Snurg worries: if a malicious (non root) user wants to hang the system he can, simply by allocating a lot of memory and writing garbadge into it.

5] my conclusion is, it is imperative to limit all non root users "memoryuse". I don't know if it is possible but even better would be if it was possible to reserve some memory out only for root to use. So if system go into stress, root can still log in and kill who needs to be killed.
 

Attachments

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#54
Just got up and had my coffee.
Nicola Mingotti exactly pointed out a thing what I am worrying about.

A malicious drive-by javascript could just eat up memory and potentially kill the desktop system.
(As I am allergic against Javascript, I use it only when I have no alternative, say, for web pages. So I am not fluent in it.
Maybe somebody else could write a memeater script to check out what happens when such a page gets called. A local file:// page would suffice for that...)

Other issue is that running embedded systems without memory buffer stop it is just crazy and irresponsible, if any serious things depend on them to work reliably.
This gets even more aggravated if they are connected to a IoT network.
Furthermore, embedded things usually have no swap device. On these, it must be possible to turn off swap entirely and replace it with sensible settings restricting memory usage by, for example, rctl.

Another consequence of this is that one probably has to add a toor user for FreeBSD, who has all rights as root, except of some memory usage restrictions.

And now I am going to play with my test PC to find out myself how totally disabling swap could be achieved in a safe manner. Will report back...
 

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#55
Just played around a bit to see how FreeBSD handles swap. Set up the test PC (40GB RAM, 550GB swap partition, of which FreeBSD apparently uses only 256GB) and played a bit with my quickndirty swapstresser utility to see how the system behaves when memory gets used.
Code:
% ./swapstresser
swapstresser: Unixoid out-of-memory situation evaluation tool
Usage: swapstresser [-bmdafstrc]
  -b   block size in megabytes to malloc every time (default 1)
  -m   max block count to allocate
  -d   cycle duration in milliseconds
  -a   allocate new block every n cycles
  -f   free a block every n cycles
  -s   if specified, stress memory every nth cycle,
       by accessing each allocated 4KiB page of chosen blocks
  -t   fraction of contiguous blocks to stress (1 = all)
  -r   if given, run this number of cycles, otherwise run infinitely
  -c   continue if allocation failed
%
If you want to play with it, download the source and its makefile from Github.

For example, when I allocate memory, I see that this memory usage does not show up in top, until it actually has been accessed. (use sufficiently long cycle times to have time to check this). Nicola Mingotti's according observation is correct. It is easily recognizable by the time it takes when the memory gets first accessed in "stressing" cycles, as these are much slower than the subsequent "stressing" cycles (when the memory is actually allocated and in use).

What I also do not understand yet is, why top-displayed RESident memory grows until it hits the full available RAM and then drops to zero, but swapctl -hgl only displays swap usage of what exceeds actual physical memory. I.e. the latter displays about 10GB swap usage when swapstresser uses 50GB, even when displayed RESident memory hits zero.

From reading the handbook chapter on security restraints, it looks like that restricting memory usage per user seems only possible by confining the user to a jail, as it seems to be possible to restrict memory usage per-jail.
This in return means that programs on embedded hardware must be run in jails, if one wants to have some sort of buffer stop protection.


However, I am still having difficulties to get rctl work like the handbook and man page suggests.
There are only very few mentions in the forums, creating the impression that rctl might be still not very widely known and used. (For example I didn't know about it until Nicola Mingotti mentioned it)
Maybe the default [PMAN=]login.conf[/PMAN] must be changed for making rctl() work?


Edit:
As a compromise for unjailed desktop usage, it then would be interesting to find out how to restrict the OS that way, that swapping is done only when all physical memory has been used up.
This way one could avert disaster when swapping sets on, initiated possibly by a mere drive-by malicious ad on a web site.
Then one could save the computer from imploding by using the ctrl-alt-bksp handbrake to zap the X server...


Edit 2:
The stroke-out statements seem to be incorrect, see my next post below.
 

Nicola Mingotti

Well-Known Member

Thanks: 123
Messages: 312

#56
Yeah, the jail ! Great Snurg, I guess you discovered the right way to go !

-] I guess I can easily write a memEater running in web browser, in Javascript, no problem.

-] about rctl, yes, to have it working it is necessary to put in "/boot/loader.conf"
"kern.racct.enable=1" .

-] An example of swapless system by default. I recently installed a FreeBSD-11.1 into a BeagleBone Black to use it as a wirelss access point. Well, that system by default is swapless [no swap in "top"] ! (i did not make it like that, I chose the default config when i installed AFAIR)

-] For ronaldlees , I tryed to read the page you found but it is really techincal, I can understand ~ 50% of it. I guess I need to read those few chapters of "The design and implementation of the FreeBSD o.s." before it.

What's next ? The next things I will try are the "jail way", I will write a JS memEater and I will try Snurg "swapstresser". I guess i can do it in the weekend.
 

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#57
What is strange now:
I had set kern.racct.enable set according to the warning I got when I ran rctl for the first time.

There seems to be either some misunderstanding on my side, or the documentation is outdated...
I tested with multiple instances of swapstresser with the -c option activated, and they together do not exceed the memory usage I set in the setting shown here.
(I apologize for any confusion my misunderstanding caused.)

Thus it does not seem to be necessary to use jails to make per-user memory restriction work...

Will have to add SHM support for swapstresser later today to test whether this also applies to SHM.
If this is the case, I'd be very happy, as it won't be necessary to make the system safe against excessive memory usage without having to use jail.
 

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#59
OT:
Is it bothersome to use/configure "jail" ? I never tried.
Not really, when you have understood the concept and are good in system administration, as every jail is in some way a separate FreeBSD installation.

The handbook is a bit outdated, it describes the now-deprecated rc.conf jail configuration. And it shows that there is a bit of work involved.
I recommend to read jail(8) and jail.conf(5) as next, they describe the modern methods.
After you have built the base jail, it's easy creating more jails, some copying, setting links and a bit configuration file editing. Thus I think it's no mistake to try the handbook setup first, and use jail.conf if one want to learn/understand dealing with the jails themselves and not with a wrapper...

Many people prefer to use jail wrappers like iocage or ezjail. However, in my impression these are not easier/quicker to grasp, the learning curve seems practically the same. They kind of make you dependent of them, as they have their own configuration files. And, there is some extra work involved which is not described in these docs, like setting up PF etc, and the device+socket communication stuff which desktop apps need, which you have to do by hand anyway.
And, sadly I am too retarded to understand the jail managers' documentation. it was much easier for me understanding the handbook and the man pages.

This all is the reason why I have the impression that only a few gurus in this forum seem to use jails for their desktop apps.

As I am no guru but instead a lousy sysadmin, I want the PF handling automated also, and so I am working on a simple-to-use jail manager that does use only jail.conf and offers an optional curses-based UI in addition to the command line. The last four weeks I have been busy with other things, though. I needed some change. Playing a bit with C was really refreshing after doing only Perl for months :) After the thing with removing swap use from my computers I will continue with that project :)
 
Last edited by a moderator:

Eric A. Borisch

Well-Known Member

Thanks: 214
Messages: 346

#61
Here's my little interactive python memory eater...
  • * doubles memory use
  • / halves
  • + increases by 10%
  • - decreases by 10%
Ctrl-c to exit.

Python:
#!/usr/bin/env python

import curses
import sys

tmem = 1

curses.initscr()
curses.savetty()
curses.raw()
curses.cbreak()

try:
    while True:
        s = 'x'*tmem
        sys.stdout.write(" {:<32d}\r".format(tmem))
        prev = tmem
        while prev == tmem:
            res = sys.stdin.read(1)
            if res == '*':
                tmem *= 2
            elif res == '+':
                tmem += tmem // 10
            elif res == '-':
                tmem -= tmem // 10
            elif res == '/':
                tmem = max(1, tmem // 2)
        s = None
except KeyboardInterrupt:
    curses.resetty()
It does not try to force memory to stay active after allocation, so you will see (once you start to build some pressure) allocation followed by swapping out, and then it should settle (swapped) until you change it again.

Edit:NOTE: This is intended for stressing memory handling on your system and can cause crashes/freezes. Do not use on a production system.
 

Nicola Mingotti

Well-Known Member

Thanks: 123
Messages: 312

#62
I wrote a JavaScript memory allocator to play with. Browsers however have their own memory management and thus may kill the offending page before it can crash the whole system.

https://five.ml/lab/allocm.html
Well done ! I tested it on my system and it works.

I was able to hang X running it. I did not wait to much and I was able to open a console out of X and kill Firefox, so I could recover the system.

I attach picture of my system hanged. There you can see all the Swap is gone and the firefox process eating all.
I guess this is the demo case Snurg was waiting: a malicious web application can hang the system.

Observe that I don't receive any complaints from Firefox, and FreeBSD (by default) does not try to kill the process even if it eats all out.

How do we procted from this is still work in progress;)
 

Attachments

swegen

Member

Thanks: 51
Messages: 75

#63
At least in your case the kernel kept a small amount of free memory so the kernel itself didn't run out of memory. But it might have if you had tried to allocate even more memory.

I have also tested these low-memory scenarios with a C program. When the available memory becomes too cramped the kernel seems to start randomly killing processes and this is the point where the system may no longer recover without a hard reset.

Recovery is much more likely if the swapspace resides in a freebsd-swap partition instead of a ZVOL.
 

Nicola Mingotti

Well-Known Member

Thanks: 123
Messages: 312

#64
At least in your case the kernel kept a small amount of free memory so the kernel itself didn't run out of memory. But it might have if you had tried to allocate even more memory.

I have also tested these low-memory scenarios with a C program. When the available memory becomes too cramped the kernel seems to start randomly killing processes and this is the point where the system may no longer recover without a hard reset.
Actually I killd Firefox when it was still allocating stuff, more it goes near to the limit of eating all memory more it becomes slow in allocating. So, I knew I had an handfull of seconds available from X first cranks to total machine hang and random killing of processes.

I also tested in C, and in Ruby as well, Eric A. Borisch posted one in Python ... it is fun to write a memory allocator;)

CAVEAT. Maybe you all know already, but, don't arrive at the limit of eating all memory on a production machine. You can make a big mess into the disk. Try this always in expendable machines.
 

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#65
Thank you very much swegen, Eric A. Borisch and Nicola Mingotti for your great input!
In particular, the Javascript allocator is a jewel... good to learn more about Javascript (I should know more about it, for web page stuff...).
(I am not intending to abuse it for crashing client's machines, but will use it to see how different browsers behave :) )

With the settings I posted in this thread, I found out that the settings apparently seem to work actually per user. It didn't exceed the memory usage, just failed to allocate memory. (tested with -c option using multiple login sessions and conventional and shared memory combined). So either I misinterpreted the manual pages and the handbook or the information there is actually incorrect.

After doing some more tests with swegen's and Eric A. Borisch's memory allocator/stressers on a throwaway installation, I think I'll be confident enough to test this soon on my main PC (which is still on FreeBSD) whether this will actually protect that swapless machines from easily being crashed.
(I had a bad accident a few years ago when I had in the file manager a lot of files marked for copying, and accidentally hit enter. The computer then attempted to opened hundreds of applications, each one for a file, HDD LED became constantly lit and it became unresponsive and had to be reset. Afterwards filesystem showed a lot of errors, but apparently no data lost.)

By the way, when I added shared memory options to swapstresser.c and did some testing, I found out that the defaults for shared memory are quite low for contemporary software use. That could have been the main cause for my problem with KDE a while back ago...)
If you want to test shared memory, too, please download the updated version from above github link. (Btw, There is still at least one bug in the app, it crashed once when I started it when no memory was left to allocate)
When you do so, you'll want to use ipcs() and ipcrm() commands to examine shared memory usage and maybe clean it up afterwards.

Have a nice weekend :)

P.S.: Eric A. Borisch, can I use any Python version, or do I need to install a particular version for your script? (Please excuse that stupid question, I have practically no Python knowledge...)
 

Eric A. Borisch

Well-Known Member

Thanks: 214
Messages: 346

#66
P.S.: Eric A. Borisch, can I use any Python version, or do I need to install a particular version for your script? (Please excuse that stupid question, I have practically no Python knowledge...)
It should work on 2 or 3. To be pedantic, you can either mark it executable and run it directly (will grab python in PATH) or pass it as the first argument to your python executable of choice.
 

swegen

Member

Thanks: 51
Messages: 75

#67
In particular, the Javascript allocator is a jewel... good to learn more about Javascript (I should know more about it, for web page stuff...).
(I am not intending to abuse it for crashing client's machines, but will use it to see how different browsers behave :) )
Thank you. I tried to write it in a self-explanatory manner and included comments in the source to make out what it does. It's easy to test mobile browsers with it too.

I noticed the tuning(7) manpage has explained some sysctl settings regarding to swap and memory usage. Some of those could be worth testing to see if they improve low-memory conditions.
 

lefsha

Member

Thanks: 3
Messages: 32

#68
Few people including me believe, that SWAP is an evil. It's like buying house you don't have money for
or like to paint old VW with gold color pretending you are rich.

The simple truth is, that at some point in life you have to be honest to yourself, when memory is over.
With or without SWAP. Using swap you mostly downgrade your PC by huge factor.

It's only acceptable if app in swap is mainly sleeping ie not in use. It's better to kill it or stop it.
If just 2 apps are concurrently running and do really need memory you end up
with constant load/unload swap pages effectively making PC unusable.

Your PC's memory is your budget - live according your budget. If it's not enough - buy more memory
or limit your usage.

Just a reminder. RAM is >10x slower than CPU. SSD is >100 slower, than CPU. HDD >1000 times slower.
Plus add huge HDD latency, where you wait 10+ms just to start reading your data.

Think of developers who optimize CPU 1st level cache usage to make your program running faster and now
you put your data into SWAP... Think of production grade In-Memory Databases broadly available.

...No wonder the new generation is writing programs in Python. Users do not care.

You need SWAP? - Consider to buy Abacus.

P.S. _Modern_ computers and SWAP is a pure joke. It did only make sense 20+ years ago.

P.P.S. I do not use SWAP more than 15 years. No single issue because of that.
 

Nicola Mingotti

Well-Known Member

Thanks: 123
Messages: 312

#69
swegen, "tuning" man has entered my "FreeBSD notes" file a couple of weeks ago;)

I can not work on this right now, I used the weekend to attach a nasty^3
network problem it was necessary to solve.

I propose we could start to write a document to summarize our advancements, or new people will come and will try to re-do all experiments we already made during the weeks.

Do you have any suggestion ? I use a lot Google Documents (requires a gmail account). Other tools for groups editing ? I am not a fan of wiki(s), they are nore a pleasure to edit them.
 

achix

Well-Known Member

Thanks: 46
Messages: 288

#70
Hey I got reading to do, you guys have been doing a great job.
Just wanted to say that after 1 week, about 4GB or 6GB out of total 8GB are used (depending on whether this akonadi_notes_agent eats up memory or not).

Talking about Server vs Desktop swapiness, whats the take of PCBSD (now TrueOS) ?
 

OJ

Daemon

Thanks: 253
Messages: 1,038

#71
Your PC's memory is your budget - live according your budget. If it's not enough - buy more memory or limit your usage.
I think you misunderstand some of the uses of swap - post 1998. :) It is not just to replace ram if there is not enough. In fact I think that is rarely what it is used for any more. It has many other purposes.
 

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#72
For some reason the rctl vmemoryuse user limiting settings seems not to cooperate with xorg. It always fails to run xkbcomp then.
And sadly with no restriction on vmemoryuse programs are not restricted to waste memory as they like.

I suspect that the reason is that the racct/rctl stuff was paid work, sponsored by the FreeBSD foundation, for usage on servers to avoid them thrashing swap, resulting in bad response times.
On servers swapping out is usually somewhat restricting memory usage, because at some point the timeout sets in, resulting the threads (and their memory) getting discarded just because of disk activity blocking processes. (You know when you have sometimes to wait looong for a page to respond, or finally failing)
So it looks to me that the racct/rctl might have primarily been not intended for desktop usage.

If you can confirm my tests result, X always failing to start even if sufficiently high limit set on vmemoryuse (I tested with these values at 10GB), what do you think about making a PR?
 

lefsha

Member

Thanks: 3
Messages: 32

#73
I think you misunderstand some of the uses of swap - post 1998. :) It is not just to replace ram if there is not enough. In fact I think that is rarely what it is used for any more. It has many other purposes.
You might be joking Mr. OJ. Am I right?
Many other purposes for which RAM is not good enough??? WOW! Tell me that secret. I promise not to tell any one else.

P.S. Well, unfortunately I know the secret... The sole purpose of SWAP is to get more money from the customer pocket.
99% people on earth have no clue how computers work. The rest 1% makes money with that business.
I am from semiconductor industry Mr. OJ. So you might guess to what percentage I do belong.

N.B. The proof, that SWAP is not necessary can be done purely mathematically with very simple assumptions.
 

forquare

Active Member

Thanks: 84
Messages: 206

#74
You might be joking Mr. OJ. Am I right?
Many other purposes for which RAM is not good enough??? WOW! Tell me that secret. I promise not to tell any one else.
Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?
 
Thanks: OJ

Snurg

Aspiring Daemon

Thanks: 327
Messages: 794

#75
Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?
By default, yes. But it's a joke. Unless you are interested in debugging the dumpfile.
And if you are, the default size of 2GB swap is a joke, too.

If you are not interested in jokes, just set dumpdev="NO" in rc.conf.

Edit: Another failed joke I still do not understand is why Xorg refuses to start, messaging me it cannot run xkbcomp when I have allocated it a maximum of 10GB virtual memory when there are still about 45GB physical memory (i.e. RAM) free.
 
Top