Clear encrypted SWAP memory?

[Crivens]

I suspect that there is still a reference to the .eli in rc.conf, so the system tries to swap to a device that is not mounted. That would explain why it is not used.

 

[grahamperrin]

Seems like removing the ".eli" solved the issue. …

With respect: not really, because the issue was not well-defined.

From the opening post:

… whenever the machine goes to sleep (S3) - the contents are dumped into swap, however, upon waking up - the swap is always full …

To me, neither of those suppositions/observations make sense.

 

[mer]

With respect: not really, because the issue was not well-defined.

An aspect may have not been well-defined. I posited that as a data point, "what is the behavior when swap is NOT encrypted", the tested hypothesis was "Does encrypting swap instigate/exacerbate the bad behavior". By eliminating encrypted swap and maintaining everything else the same (workload, suspend behavior) if it was better, then encrypted swap was a factor in the issue.

To me, neither of those suppositions/observations make sense.

Based solely on the OP, agree. But reading through the rest of the questions and responses further in the thread, the fact that swap was encrypted with a one time key was interesting to me.
My thought process was:
swap encrypted with one time key, laptop suspended, what happens to that key?
Is that key maintained across unsuspend? If not, then swap can be tagged in use but never cleared. If so, then swap should clear.

My simple question of "what happens if swap is not encrypted and you do everything else the same" was to get data points. But the OP's reports of "I removed the .eli so not ecrypted and not using swap anymore" is boggling.

 

[grahamperrin]

Is that key maintained across unsuspend?

I assume so.

 

[mer]

I assume so.

Me too, but the observed behaviour implies "maybe not"

 

[Tracker]

. on Thinkpads, Fn + F4

Somehow this doesn't work for me. I remember these function buttons were a pain to get to work correctly last time I had tried.

Also tried to add my user name to operator group but seems like it won't permit - maybe it requires a restart?
> sudo pw group mod operator -m myusername

That is very odd.
So you edited /etc/fstab, removed the .eli, rebooted, did your typical workload, suspended, etc and it's not using swap now?

Very strange.

To be specific : commented out the same line with ".eli" and copied the same without the extension. Strange indeed.

[FONT=monospace]Tracker[/FONT]: you quote me as saying just

zzz

Please don't get in the bad habit of quoting a word or phrase out of context, when I said:

Sorry - I thought the reference to the text was obvious - didn't mean to imply you were saying something else.

Why assume a 'shortcut', let alone a shell alias? Whenever in doubt, ask man (or apropos):
zzz(8)

Interesting - didn't know about zzz at all.

I suspect that there is still a reference to the .eli in rc.conf, so the system tries to swap to a device that is not mounted. That would explain why it is not used.

cat /etc/rc.conf | grep eli doesn't return anything.

To me, neither of those suppositions/observations make sense.

is boggling.

Yes, I didn't suspect simply removing the extension would solve the issue as well. Can't make sense of it myself. But I have now, multiple times, closed the lid and tried `sudo acpiconf -s 3` - the swap space isn't being utilized at all. I don't recall doing much else to solve this issue.

For the sake of curiosity - would it be worthwhile to enable the ".eli" extension in fstab and checking if swap comes back into use? Perhaps not now, but maybe sometime in the next few days. Might throw some light if that indeed was the, seemingly strange, culprit.

 

[grahamperrin]

culprit

Another gentle reminder:

Basics. … diagnose …

For definite, useful information about how swap is used, run:

systat -swap

…

Do not disable swap.

Has there been an opportunity?

Please, do diagnose things.

Discover whether what's truly in swap matches assumptions about what's there (or why it's there).

 

[smithi]

My thought process was:
swap encrypted with one time key, laptop suspended, what happens to that key?

If the key is stored in RAM and all RAM is preserved by definition (refreshed if dynamic) in suspend state, it must be there upon resume.

All fixed disk is likewise preserved, including swap, though swapspace via USB may not be so trustworthy.

Is that key maintained across unsuspend? If not, then swap can be tagged in use but never cleared. If so, then swap should clear.

You'd think so, but I don't know.

My simple question of "what happens if swap is not encrypted and you do everything else the same" was to get data points. But the OP's reports of "I removed the .eli so not ecrypted and not using swap anymore" is boggling.

I'm getting used to boggledness.

 

[smithi]

>> . on Thinkpads, Fn + F4

Somehow this doesn't work for me. I remember these function buttons were a pain to get to work correctly last time I had tried.

Fn+F4 is hard wired, like the power button. There may be one way to disable or redefine it, via devd.

Please show in a code block:
% sysctl dev.acpi_ibm

Also tried to add my user name to operator group but seems like it won't permit - maybe it requires a restart?
> sudo pw group mod operator -m myusername

Syntax error: pw(8)

 

[Crivens]

Fn+F4 is hard wired, like the power button. There may be one way to disable or redefine it, via devd.

Not really. I had put haiku on old ThinkPads I have and these buttons don't work there.

 

[mer]

I'm getting used to boggledness.

Unfortunately, so am I.

"For the sake of curiosity - would it be worthwhile to enable the ".eli" extension in fstab and checking if swap comes back into use? Perhaps not now, but maybe sometime in the next few days. Might throw some light if that indeed was the, seemingly strange, culprit."

I've been running with .eli on my swap partitions for a while, across 13.x now 14.0-RELEASE, on 2 different systems. One having 32GB the other 8GB of memory and I've never seen swap used. Both are desktops so I'm not suspending like you are so I'm guessing it may be the combination of encrypted swap and suspending that is somehow the root cause. The only way I can wrap all that up is the onetime key not getting preserved across the suspend operation so the swap can't be cleared.

 

[smithi]

It wasn't me, but Tracker who said:

"For the sake of curiosity - would it be worthwhile to enable the ".eli" extension in fstab and checking if swap comes back into use? Perhaps not now, but maybe sometime in the next few days. Might throw some light if that indeed was the, seemingly strange, culprit."

I've been running with .eli on my swap partitions for a while, across 13.x now 14.0-RELEASE, on 2 different systems.

I think Tracker may only have this one Thinkpad T480, suggesting reticence to reboot often to follow suggestions ...

One having 32GB the other 8GB of memory and I've never seen swap used. Both are desktops so I'm not suspending like you are so I'm guessing it may be the combination of encrypted swap and suspending that is somehow the root cause. The only way I can wrap all that up is the onetime key not getting preserved across the suspend operation so the swap can't be cleared.

Well, we don't really know that the swap was being written on suspend, that may be another supposition. Perhaps it's as or more likely swap being filled occurred just after resume, as just before suspend?

It's a mystery, but short of a proper PR process to get relevant eyes onto it, I'll be surprised if it's solvable here.

Meanwhile waiting for progress on an apparently unrelated bogglement:
https://forums.freebsd.org/threads/...-of-freebsd-like-in-windows.92207/post-645953

 

[grahamperrin]

… short of a proper PR process to get relevant eyes onto it, I'll be surprised if it's solvable here. …

There's not enough for a bug report, use of Bugzilla for general discussion is discouraged …

 

[Tracker]

Update : Still haven't restarted the machine. Just noticed swap is now finally being utilized (via htop) - although much lesser (about 1.72GB / 8 GB as of now) - still have unused 40 GB of RAM, not sure why/when it's requiring the swap.

Sorry for the confusion here - the swap usage for 0 until yesterday.

Fwiw - it took considerable time for the swap to be utilized this time (few days) and much lesser usage(used to be full earlier). Earlier (before unencrypting the swap) the usage used to show up much faster.

 

[mer]

Update : Still haven't restarted the machine. Just noticed swap is now finally being utilized (via htop) - although much lesser (about 1.72GB / 8 GB as of now) - still have unused 40 GB of RAM, not sure why/when it's requiring the swap.

Sorry for the confusion here - the swap usage for 0 until yesterday.

Fwiw - it took considerable time for the swap to be utilized this time (few days) and much lesser usage(used to be full earlier). Earlier (before unencrypting the swap) the usage used to show up much faster.

And you've been doing the same workload as before? work work work, suspend, unsuspend, work work work?

I wonder if this is some data in support of encrypt key not being supported across suspend/unsuspend.
encrypted swap, one time key:
suspend, put stuff on swap with key A
unsuspend, key A is "lost" so swap is not cleared, but swap code knows space is in use and maybe now we have key B
suspend using key B, unsuspend now key B is lost, swap has stuff from key A and key B.

That almost sounds like the symptoms

 

[grahamperrin]

not sure why/when it's requiring the swap.

systat -swap

 

[smithi]

I wonder if this is some data in support of encrypt key not being supported across suspend/unsuspend.

Let's call them suspend and resume, aligned with system use in /etc/rc.{suspend,resume} as referenced in /etc/devd.conf.

encrypted swap, one time key:
suspend, put stuff on swap with key A

That would imply that use of geli for swap sets up a devd event/s to intervene in the suspend/resume process. Can you confirm that it does that?

unsuspend, key A is "lost" so swap is not cleared, but swap code knows space is in use and maybe now we have key B
suspend using key B, unsuspend now key B is lost, swap has stuff from key A and key B.

That's pretty speculative; unless geli swap code checks that "something like" some timestamp hashed with the key has changed on each and every access, we know RAM and disk is preserved.

And why would it want to waste so much time? S/R is supposed to be transparent to all processes, except the small amount of memory used to store state information.

That almost sounds like the symptoms

If we're hearing the full story, maybe ... but unless there's some sign of geli-related devd events, how is it possible?

 

[Tracker]

And you've been doing the same workload as before? work work work, suspend, unsuspend, work work work?

Yes - only suspend,resume. To be more specific I have a (large) instance of chromium and another browser running some jupyter notebooks (among other things in the background tabs) .... besides that a few terminals open, not doing much at all there.

systat -swap

                   /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
ada0p4           7932M 1949M XXXXXXXXXXXX

Pid    Username   Command     Swap/Total Per-Process    Per-System

(no pids,username,command, etc show under that)

 

[mer]

That's pretty speculative;

Yep, absolutely. I run desktops, don't do suspend/resume so I'm thinking out loud on this.
What I know about the swap stuff is simply adding a .eli to the partition definition /etc/fstab makes the swap code do one time encryption.
My understanding of that is if you have stuff written to swap, then you shutdown and reboot, you get a different key and the contents of swap is unreadable because the original key is gone.

Perhaps my hypothesis is wrong, just trying to figure out possibilities based on what we've been told.
What we've been told is the user is keeping the workload/pattern the same, the only thing different is ".eli on swap partition or not"

 

[grahamperrin]

(no pids,username,command, etc show under that)

Thanks. It'll be useful to capture output at a time when a process is identifiable.

So, please re-run the command at an appropriate time.

I should not leave it running constantly (in my case, doing so impacts performance; YMMV).

 

[smithi]

Yep, absolutely. I run desktops, don't do suspend/resume so I'm thinking out loud on this.
What I know about the swap stuff is simply adding a .eli to the partition definition /etc/fstab makes the swap code do one time encryption.

Ok, well my knowledge of geli could be seen to match yours of suspend / resume ... flawed or minimal in my case ...

My understanding of that is if you have stuff written to swap, then you shutdown and reboot, you get a different key and the contents of swap is unreadable because the original key is gone.

Sure, but there's no similarity between shutdown / reboot and suspend / resume, given that all RAM (apart from a small, specific state save area) and all directly attached disk is preserved by S3 suspend.

I.e. S/R must be transparent to all processes, with the possible exception of a process that (I postulate) deliberately updates and checks stored timestamps with the specific intent of detecting "missing time" from having been suspended!

Perhaps my hypothesis is wrong, just trying to figure out possibilities based on what we've been told.

No shortage of guesswork ;-)

What we've been told is the user is keeping the workload/pattern the same, the only thing different is ".eli on swap partition or not"

It would be useful knowing if geli-encrypted swap has a real and consistent problem with being suspended, but it's not something I can test. This is why I suggested a PR, so those who should know can rule it in or out ...

 

[Tracker]

UPDATE: Swap almost became full now: 7.71 GB full / 7.75GB available

Thanks. It'll be useful to capture output at a time when a process is identifiable.

So, please re-run the command at an appropriate time.

Guess what - tried running the same command at almost full swap. Still nothing :/ (no pids, username,command etc)

systat -swap

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||||||||||||||||||||||||||||||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
ada0p4           7932M 7692M XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Pid    Username   Command     Swap/Total Per-Process    Per-System

Thankfully saved the incognito tabs i'm used to working with.

Starting to wonder if this could be due to Chromium acting weird with A LOT of tabs in the background? But then it should have showed up under systat command, wouldn't it?

Still confused what is causing this (plenty of unused RAM)

 

[Crivens]

This could be a memory leak in some browser component. The memory which is allocated and written to, but is not touched for some time, gets to be written to swap "just in case" it needs to be recycled in short time. The the kernel can drop the page at once, as it is already in sync with the swap space. This even is done with free memory around. So a trickeling memory leak could slowly fill up the swap space, but that would be connected with a PID. Is it cleaned up when you restart the browser?

 

[grahamperrin]

In this example, Firefox running uses a tiny amount of swap.

Note the Mem and Swap indicators at the foot of GKrellM:

 

[grahamperrin]

Still nothing :/

It may take some time for window content to appear entirely as it should. The waiting period, after resizing a window, may be longer if the system is noticeably busy.

That said, I don't recall ever seeing an amount used (i.e.more than zero) with zero processes listed.

I should not leave it running constantly (in my case, doing so impacts performance; YMMV).

That, and the waiting period, are not ideal. Technically, I don't know the reason(s) … in simple terms, I guess there's a balance to be struck. From devstat(9) DESCRIPTION:

… reasonably detailed statistics while utilizing a minimum amount of CPU time to record them. Thus, no statistical calculations are actually performed in the kernel portion of the devstat code. Instead, that is left for user programs to handle. …

Two screenshots below. I might add another one, or two, later.