• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

swap - memory

lefsha

Member

Thanks: 3
Messages: 32

#76
Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?
1. No one knows upfront, what causes the kernel panic. It might be well an issue with a hard drive.

2. If a hard drive IS working, nothing prevents any program including kernel to dump RAM on disk in a form of file, which can be easier recognized by admin, in case the machine does have an admin attached to that particular computer. On most (99%) of user laptops it is not the case.
The same can be said about any embedded things. So there is nobody who will look at it. And even fewer people are able to understand what it means.

3. Any causal program which is running on a computer from time to time saves its data on disk. None of them requires swap for that purpose.

4. Last but not least. What does word "swap" means? It doesn't mean a permanent data storage, neither a database and nothing else as well.
It does mean swap - an outsource of memory pages based on certain algorithm to another storage space, which is a hard drive for example!
The same page goes back into memory once there is enough room for it OR (unfortunately) it is required for a running program.

Nothing prevents any one to save his favorite movie on swap partition. Nothing prevents a man to shave with a knife. And many did in the past. But it is a wrong usage of the instrument which has been developed for a different purpose.

Am I wrong?
 

lefsha

Member

Thanks: 3
Messages: 32

#78
I don't think you are wrong, but you asked what the other uses for SWAP were, so I gave you one. It may not be a useful one to you, or to many other people, but it is a use :)
If you understood me correctly my message was, that using SWAP for something else means using a wrong tool for a different purpose it was not made for. I could easily create a swap partition and use it for database server as raw storage. It is no way different than your proposal.
It has nothing to do with swap. I could go further, create a swap partition and mkfs on it. That would be my secret storage :cool: everyone else would think it is a swap, but not :p it's a special thing... I tell nobody about. I could write a simple program to save on swap partition a blockchain sequence and that way excuse the huge size of that space. If you are creative there may be plenty of possibilities to use swap for something it was not made for.

What I have said at the very beginning is - SWAP as swap is not necessary, more over it is a bad idea, bad instrument serving no purpose.
I can't discuss multiple other crazy ideas what people might use that partition for. It's actually off-topic.

By other usage I was thinking of something related to deal with not sufficient RAM. Instead it's just another security hole. By getting access to swap, I could manage what content will be loaded into memory and... possibly executed. Moreover my virus could survive the reboot...
 

Snurg

Aspiring Daemon

Thanks: 324
Messages: 795

#79
That would be my secret storage :cool: (...) It's actually off-topic.
Actually swap, kernel and application dumps are highly interesting forensic stuff.
You might find sweet things inside them the user didn't intend at all to make strings-able for "others".

So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
And, if you are really into swapping, turn encryption on.
 

lefsha

Member

Thanks: 3
Messages: 32

#80
Actually swap, kernel and application dumps are highly interesting forensic stuff.
You might find sweet things inside them the user didn't intend at all to make strings-able for "others".

So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
And, if you are really into swapping, turn encryption on.
Exactly.
 

gnath

Member

Thanks: 16
Messages: 62

#82
So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
And, if you are really into swapping, turn encryption on.
I as a user turn off core dumps. But SirDice and others said low use of swap in normal use. In early days we used to set swap partition as twice of memory. Now probably we can add swap file as required. Here in freebsd there hardly any surge requirement of swap (my case). Initially there was no swap partition witout any problem. Some body said that physical partition required by OS. I added linux partition for freebsd. But noticed never used during compilation.
I do not know swap allocation is monitored & done by only OS or the running programs/process or both. Freebsd's
foot print is very low Firefox require proportionate to open tabs and also open instances. I think running no. of process
as required will be guiding factor for swap for a particular OS.
 

Snurg

Aspiring Daemon

Thanks: 324
Messages: 795

#83
Talking about Server vs Desktop swapiness, whats the take of PCBSD (now TrueOS) ?
But SirDice and others said low use of swap in normal use.
Yes. And I find it highly annoying when there is always a small latency because there is some swap access when I switch windows, screens or the like.
Dunno whether there is some configuration on PCBSD resp. TrueOS which avoids this annoying latency.
I mean, swapping out is just idiotic when >40GB RAM are shown unused by top.

Anyway, back to the experiments on setting limits to memory usage:

Code:
user:snurg:memoryuse:deny=10g/user
# if swapuse is set to 0, then user cannot log in anymore
# if it is set to 1M one at least gets "out of memory" message
# if it is set to 1G X fails to start
# user:snurg:swapuse:deny=1G/user
# after disabling swapuse option X still does not start.
# So trying to disable the last three parms:
#user:snurg:shmsize:deny=2g/user
#user:snurg:datasize:deny=3g/user
#user:snurg:stacksize:deny=3g/user
# finally I found that X does not start when this option is set, no matter how high the value. Isn't this crazy?
#user:snurg:vmemoryuse:deny=10g/user
When using no X, then these settings work fine and get honored. (except swapuse, but when there is no swap, it cannot be used anyway, so probably no need to set it)
 

achix

Well-Known Member

Thanks: 46
Messages: 285

#84
After one day swap seems stabilized. 1.7GB out of total 8GB. Seems like the previous version of KDE had some nasty mem leaks.
 

Nicola Mingotti

Active Member

Thanks: 89
Messages: 185

#86
I add a reason to run FreeBSD in "swap-off", which I just discovered.

It is a few days that I am not switching off FreeBSD at night,
there are too many windows and desktops open. Re-open, Re-position
all of them on the new day would be a bit boring. So I run a screenlock and
let the system run all night.

On the next day, what I see is that, when I log into the
screenlocker, which is snappy, I find an extremely slow
system. After 5-10 minutes of use the speed becomes regular.

I guess (I don't have data on this by now) that at night
FreeBSD is moving basically all my programs to the swap.

ronaldlees already pointed out in this thread that FreeBSD puts unused stuff in swap
proactively, to be ready to answer to fresh new requests. [this not a quote, i am rephrasing]
So, Its behaviour makes sense, as is it a "server with a network bias".

But, from a desktop machine perspective, there will be not
new unexpected requests, so it makes sense to turn swap off,
and have the system focus on what the main user whats to run.
 

Nicola Mingotti

Active Member

Thanks: 89
Messages: 185

#87
I went I little forward on this subject:

AIM:
1] I want to keep "swap off", but I don't want to loose data or corrupt my system.
2] I don't want to let the system do the guardian of memory limits because, when
those limits are surpassed, AFAI-get it is going to kill
programs choosing them I a way I do not understand.

METHOD.
I wrote a little script (with X GUI) that every second checks the available
free memory using top. If memory free memory goes below e.g. 1G
then the script does swapon -a, it is allowed by sudo.

PROBLEM(s).
1] What is Inactive Memory I see in top ? I do not undertand.
I found this post but it is very old:
https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/

2] Can I consider "Inactive Memory" a synoim of "dirty memory that can be re-allocated" ?

3] Is there a way I can clean "Inactive Memory" and make it "Free" ?

4] (note) I observed If I leave the computer on all night, in the morning
the amount of "Inactive Memory" is very large.
 

Crivens

Moderator
Staff member
Moderator

Thanks: 484
Messages: 1,403

#88
Let me pipe in with some info about why swap is a good idea.

You need some MB to help out when a contiguous memory area is needed but there are some mapped pages in the way. Otherwise the mapping will fail.

The kernel dumps only contain active memory, if my memory serves me right. So a dumpdev of 2GB is enough. Please note that that is not a swap device.

He who trusts a kernel not to wreck the file system after something went terribly wrong needs to take some hard lessons from fate - like finding half your files in /lost+found and containing the string "told you so".

And for Xorg, please check how much virtual address range it maps (framebuffers etc). These also count and grow very fast.

Sorry for being so short here, typing on phone screen sucks a golf ball trough a garden hose.
 

Maxnix

Well-Known Member

Thanks: 161
Messages: 283

#89
I guess (I don't have data on this by now) that at night
FreeBSD is moving basically all my programs to the swap.

@ronaldlees already pointed out in this thread that FreeBSD puts unused stuff in swap
proactively, to be ready to answer to fresh new requests. [this not a quote, i am rephrasing]
So, Its behaviour makes sense, as is it a "server with a network bias".

But, from a desktop machine perspective, there will be not
new unexpected requests, so it makes sense to turn swap off,
and have the system focus on what the main user whats to run.
You could try setting kern.ipc.shm_use_phys=1 to "cause all System V shared memory segments to be mapped to unpageable physical RAM". Look here: https://wiki.freebsd.org/SystemTuning#SYSCTL_TUNING
PROBLEM(s).
1] What is Inactive Memory I see in top ? I do not undertand.
I found this post but it is very old:
https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/

2] Can I consider "Inactive Memory" a synoim of "dirty memory that can be re-allocated" ?

3] Is there a way I can clean "Inactive Memory" and make it "Free" ?

4] (note) I observed If I leave the computer on all night, in the morning
the amount of "Inactive Memory" is very large.
Help yourself: https://wiki.freebsd.org/Memory :)
 

Nicola Mingotti

Active Member

Thanks: 89
Messages: 185

#90
Thank you for suggestions,

In short, what I propose is:
1] Turn swap on only when there is less than 1G free memory available. What danger do you see on this procedure ?

2] In case swap had to be always on I have a second option.
Could I make a memory disk and put the swap there?
 

Nicola Mingotti

Active Member

Thanks: 89
Messages: 185

#91
You could try setting kern.ipc.shm_use_phys=1 to "cause all System V shared memory segments to be mapped to unpageable physical RAM". Look here: https://wiki.freebsd.org/SystemTuning#SYSCTL_TUNING

Help yourself: https://wiki.freebsd.org/Memory:)
Thank you for the links. The first one for me is new !
I will read it after diner;)

The second link no, I know it. That is what raised my doubts actually.
What it is says contrasts with what other people told in this discussion:
https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/
 

Maxnix

Well-Known Member

Thanks: 161
Messages: 283

#92
Thank you for suggestions,

In short, what I propose is:
1] Turn swap on only when there is less than 1G free memory available. What danger do you see on this procedure ?
AFAICT no one, but you could try first setting that sysctl variable.
2] In case swap had to be always on I have a second option.
Could I make a memory disk and put the swap there?
This would be like not having swap, after all. In that case just disable it, and keep everything in RAM automatically.
The second link no, I know it. That is what raised my doubts actually.
What it is says contrasts with what other people told in this discussion:
I don't see contrast between them: Inactive memory: memory not recently used by userland (your programs), that cannot be used by other programs (I guess each program can access only it's own chunks of it if has some) or used by the filesystem (UFS only?) for caching and is swappable. But maybe someone more knowledgeable will correct me. :)
 

Nicola Mingotti

Active Member

Thanks: 89
Messages: 185

#93
Maxnix, here is the inconsistency I see between the two sources.

---- Wiki states ----
Inactive Queue
  • Inactively (not recently used) used by userland
  • Not recently used (LRU)
  • Cannot be re-allocated to other processes
-----------------------

Instead on the (old) thread Beastie says
----------------------
In inactive memory you may have clean or dirty pages. Dirty pages are no longer in use, may be removed in the near future (after being synced to disk) and reallocated for something else.
----------------------

I have red the first link you posted but I don't think my
issues with swap are strictly related to shared memory. [ to be clear, my phd is not in computer science so I may be boldly wrong here ].
----------------------- quoted, only changes in colors, bold, underline.
  • The kern.ipc.shm_use_phys sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on). Setting this parameter to 1 will cause all System V shared memory segments to be mapped to unpageable physical RAM. This feature only has an effect if you are either (A) mapping small amounts of shared memory across many (hundreds) of processes, or (B) mapping large amounts of shared memory across any number of processes. This feature allows the kernel to remove a great deal of internal memory management page-tracking overhead at the cost of wiring the shared memory into core, making it unswappable.
--------------------------------------------

P.S. Colors, bold and underline are for my convenince in finding
important parts. No "shout-effect" intended;)
 

Snurg

Aspiring Daemon

Thanks: 324
Messages: 795

#94
You need some MB to help out when a contiguous memory area is needed but there are some mapped pages in the way. Otherwise the mapping will fail.
So FreeBSD cannot just remap without a swap device??
The kernel dumps only contain active memory, if my memory serves me right. So a dumpdev of 2GB is enough. Please note that that is not a swap device.
Is the dumpdev="NO" directive unreliable? If so, can I use /dev/null for this? If not, that would be bad for embedded usage.

He who trusts a kernel not to wreck the file system after something went terribly wrong needs to take some hard lessons from fate - like finding half your files in /lost+found and containing the string "told you so".
Weren't softupdates introduced to make such less probable?

And, if the kernel wants to crash, does having a dump/swap device save the file system from being corrupted? ;)
And, if the kernel uses normally, say 5GB, and user memory usage is restricted so that the kernel can grow to, say, 10GB, will a swap of 2GB actually "save" the system instead of just postponing the crash?
Or will having a swap partition enabled (or even worse, a swap file), just make the system unresponsive while swapping, maybe creating even more mayhem by things like important filesystem metadata updating getting out of sync?

Of course, on web servers swapping will have a "protective effect" for the system by causing many threads to time out and thus freeing memory by dying off/being removed from memory... maybe this is the indirect "buffer stop effect" of having swap :)

And for Xorg, please check how much virtual address range it maps (framebuffers etc). These also count and grow very fast.
X fails to start even the user has 10 gigabytes virtual memory allocated. Apparently nobody knows why. Is the setuid X the problem or is the racct/rctl system broken?

Crivens, do you understand that the statement "you must have a swap partition" still does appears not very convincing to me?

Regarding desktop and laptop usage, I am going to ditch FreeBSD anyway, because its S3 suspend with newer Nvidia drivers meanwhile is broken beyond repair, and with systemDOS even S4 suspend (i.e. hibernate) works just fine.
Additionally the latter's swappiness can be easily configured, without touching undocumented kernel knobs.
 

Crivens

Moderator
Staff member
Moderator

Thanks: 484
Messages: 1,403

#96
Okay, now with some real keyboard...

1) Imagine this, you need some range of physical memory for some DMA activity, but there is one page in the way that is allocated and dirty. With swap, the allocator can drop that one, wait for the laundry, and allocate it when done. The allocation succeeds, but you need to wait some milliseconds. Without this, the allocation might fail. That is why other kernels have things like a memory defragmentation now. No big space is needed, and it might even be a memory disk. But that smells a lot like the duke of münchhausen to me. If the system by now can re-map on the fly - I don't know right now. It has been some time since I last dived into that area and read the source code.

2) A swap file is a last resort, the data has to go trough a file system layer also. Why? Just give it some GB of space in a partition - and maybe do that in all disks you have in the system. Interleaved swap space is better than funneling all IO trough one disk. And even with soft updates you still dice with the dataloss. There is this game where you get handed a revolver with one bullet and you point it at your server and pull the trigger. Now there are models with more than 6 chambers - how many will it take for you to accept the risk? For me, I don't play that game. I prefer the thing to fall over, give me a kernel core to see what screwed up and not mess up the file system any more than it already did.

3) Swap space and crash dump space can be separated. You can run a system without the swap space and still have a crash dump location available. And that needs to be a device, not a file. Otherwise, your swap partition will be used in that case (but not your swap files, if I remember correctly). Savecore will then pick the dump up during reboots and stash them away for later. And once you start developing drivers, this is very helpful for you. And one of the first steps in my current job was to enable core dumps in the embedded devices because when the field testers come in and complain about sporadic restarts without any idea of why - then this is what you want.
 

Gray Jack

New Member

Thanks: 7
Messages: 16

#97
There is no way to configure "swapness"? Like, define from, dunno, from 1 to 100, the level of priority to use swap partition
 

Gray Jack

New Member

Thanks: 7
Messages: 16

#99
The swappiness parameter is only for Linux as far as I know.
That's what I thought, that's why I used quotes, cause there was a very good chance that swapness concept didn't exist on FreeBSD.
But it is a great concept, at least on Desktop area, helps a bunch on responsiveness of the Desktop on Linux, it's maybe the only concept that I miss in other UNIX/UNIX-Like systems
 
Top