Clear encrypted SWAP memory?

[Tracker]

Hi,

Have a geli encrypted hard disk with encrypted swap setup - how does one clear the swap memory files?

To elaborate whenever the machine goes to sleep (S3) - the contents are dumped into swap, however, upon waking up - the swap is always full and it doesn't go back to RAM (which I have more than enough)

I have already tried the following

sudo swapoff -a 
Password:
swapoff: /dev/ada0p4.eli: Cannot allocate memory

 

[mer]

Do you have an external device, even a USB drive around?
If so have you tried creating a swap partition on it, plugging it in and doing a swapon to that just to see if it lets you swapoff the encrypted swap?
Have you tried not encrypting the swap (yes I understand you want it encrypted) to see if the behavior changes?

 

[Tracker]

Do you have an external device, even a USB drive around?
If so have you tried creating a swap partition on it, plugging it in and doing a swapon to that just to see if it lets you swapoff the encrypted swap?

Not really - would prefer the method below

Have you tried not encrypting the swap (yes I understand you want it encrypted) to see if the behavior changes?

How do I go about not encrypting it? Does it need a restart?
I tried the following but it didn't work

sudo geli detach /dev/ada0p4.eli 
geli: Cannot destroy device ada0p4.eli (error=16)

 

[mer]

Not really - would prefer the method below

How do I go about not encrypting it? Does it need a restart?
I tried the following but it didn't work

sudo geli detach /dev/ada0p4.eli
geli: Cannot destroy device ada0p4.eli (error=16)

You may need to boot into single user mode, then modify /etc/fstab. The line for swap: if you just drop the .eli extension on the device then swap will not be encrypted.

 

[Tracker]

You may need to boot into single user mode, then modify /etc/fstab. The line for swap: if you just drop the .eli extension on the device then swap will not be encrypted

Ah yes - I am trying to avoid a reboot - this problem is fixed by a reboot but I usually don't like to reboot in the middle of work. Is there some other way?

Did you mean this by dropping the .eli extension?

sudo geli detach /dev/ada0p4
geli: Cannot destroy device ada0p4.eli (error=16)

Didn't work, same error it seems

 

[mer]

Ah yes - I am trying to avoid a reboot - this problem is fixed by a reboot but I usually don't like to reboot in the middle of work. Is there some other way?

Did you mean this by dropping the .eli extension?

In the /etc/fstab you will have a line similar to this:
/dev/ada0p2.eli none swap sw 0 0

I'm saying "boot to single user, edit that line to remove the .eli extension save then continue to boot"

If you don't want to reboot, the only possible way that I can think of is physically add another swap device bigger than your current one, swapon to the new device, then I would close running applications and log out or get back to a command line and try to swapoff the original swap device. Yes it interrupts your work and it may not actually do what you want.

A theory only, based on nothing more than wild guesses from too much coffee or not enough beer: that the sleep and push to swap is maybe losing track of what the swap contents actually represent so coming out of sleep swap doesn't know it needs to be cleared or perhaps when swap is encrypted the one time key is somehow lost, so contents of swap are inaccessible but enough is unencrypted to know "swap is in use".

My datapoint of "does it happen if you do not encrypt swap" would reflect to the second part of that theory.

Other than "search the forums because I seem to recall a long thread about this exact same scenario and I don't recall if anything was discovered on how to fix it", that's all I've got.

 

[cracauer@]

Why would it dump to encrypted swap in the first place?

I might be misunderstanding something here. Pure OS-driven suspend-to-disk is a Linux thing, and it can indeed use encrypted storage for that.

 

[grahamperrin]

… with encrypted swap

… whenever the machine goes to sleep (S3) - the contents are dumped into swap, however, upon waking up - the swap is always full …

I never had that problem.

Which version of FreeBSD, exactly?

uname -aKU

How much did you give to swap?

% swapinfo -h
Device              Size     Used    Avail Capacity
/dev/ada1p2.eli      16G       0B      16G     0%
%

 

[smithi]

Have a geli encrypted hard disk with encrypted swap setup - how does one clear the swap memory files?

I know zero about geli or encrypted swap, but ...

To elaborate whenever the machine goes to sleep (S3) - the contents are dumped into swap,

Contents of what? Not main memory, it's just kept refreshed. Video memory is saved somewhere?

however, upon waking up - the swap is always full and it doesn't go back to RAM (which I have more than enough)

I can't follow this. Maybe Andriy can, or knows who to ask?

 

[mer]

Doesn't the kernel coredump to swap by default?

 

[grahamperrin]

… kernel coredump …

I do not perceive a kernel panic (system crash), I see colloquial use of the word dump:

… whenever the machine goes to sleep (S3) - the contents are dumped into swap, however, upon waking up …

– that's a wake from sleep (logically, it can not be a wake from a system crash).

---

Kernel panics​

Where swap is GELI-encrypted, crash dumps should be to the non-encrypted physical device.

Spoiler: Example: adap1.eli for swap, adap1 – not .eli – for system crash dumps

% lsblk /dev/ada1
DEVICE         MAJ:MIN SIZE TYPE                                    LABEL MOUNT
ada1             0:136 932G GPT                                         - -
  ada1p1         0:138 260M efi                              gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                           - -
  ada1p2         0:140  16G freebsd-swap                        gpt/swap0 SWAP
  ada1p2.eli     1:11   16G freebsd-swap                                - SWAP
  ada1p3         0:142 915G freebsd-zfs                          gpt/zfs0 <ZFS>
  ada1p3.eli     0:148 915G -                                           - -
  <FREE>         -:-   708K -                                           - -
% swapinfo
Device          1M-blocks     Used    Avail Capacity
/dev/ada1p2.eli     16384        0    16384     0%
% sysrc -f /etc/rc.conf dumpdev
dumpdev: /dev/ada1p2
% dumpon -l
ada1p2
%

dumpon(8) – dumpon — specify a device for crash dumps

pstat(8) – pstat, swapinfo — display system data structures

 

[larshenrikoern]

What about using swapoff -a to empty the swap (if you have enough memory. But you said you have). And afterwards swapon -a to reenable it

UPDATE: I see that was what you tried. I think the reason why it does not work is that you do not have enough memory if the swap device is bigger than the memory or actually in use. But that the swap is full should not be a problem if you have enough free memory. The swap is not written back in normal use either. But if swap is needed the content on the swapdevice is overwritten when needed. In other words do you experience any performance problems with your current configuration (besides seeing a number on top or another program)

 

[grahamperrin]

… And afterwards swapon -a …

Consider <https://github.com/Freaky/swapflush>. swapflush uses swapon(8) to remove then re-add the specified swap devices, while ensuring that they always remain open — for detach-on-last-close devices such as geli(8) onetime providers to survive the process.

I sometimes run /home/grahamperrin/dev/swapflush/swapflush -a only if I'm slowly preparing to restart the OS.

---

Note, premature flushing is generally counterproductive.

 

[Tracker]

If you don't want to reboot ....

Yes - intend to find a solution that doesn't involve reboots - since this is a recurring problem. Rebooting fixes it anyways - until it happens after some time, again.

Yes it interrupts your work and it may not actually do what you want.

Oops - hoping for something effective

A theory only, based on nothing more than wild guesses from too much coffee or not enough beer: that the sleep and push to swap is maybe losing track of what the swap contents actually represent so coming out of sleep swap doesn't know it needs to be cleared or perhaps when swap is encrypted the one time key is somehow lost, so contents of swap are inaccessible but enough is unencrypted to know "swap is in use".

My datapoint of "does it happen if you do not encrypt swap" would reflect to the second part of that theory.

Tried to switch it off - but didn't work.

Which version of FreeBSD, exactly?

Spoiler: uname -aKU

FreeBSD echo 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64 1302001 1302001

How much did you give to swap?

swapinfo -h
Device              Size     Used    Avail Capacity
/dev/ada0p4.eli     7.7G     7.7G      12K   100%

FYI - RAM is 64G - so memory isn't a constraint - currently only 15G used of 64, while htop shows swap of 8G to be at full capacity

Contents of what? Not main memory, it's just kept refreshed. Video memory is saved somewhere?

I would guess it's contents of main memory that are stored there - not that I can say with 100% confidence but didn't try anything special with Video memory

that's a wake from sleep (logically, it can not be a wake from a system crash).

Yes - wake from sleep, a system crash/reboot solves it naturally.

UPDATE: I see that was what you tried. I think the reason why it does not work is that you do not have enough memory if the swap device is bigger than the memory or actually in use. But that the swap is full should not be a problem if you have enough free memory

Enough RAM/free memory - see above 15G of 64G used.

Consider <https://github.com/Freaky/swapflush>. swapflush uses swapon(8) to remove then re-add the specified swap devices, while ensuring that they always remain open — for detach-on-last-close devices such as geli(8) onetime providers to survive the process.

Not sure I have tried such a thing before - are there no steps to simply follow and do it on my machine instead of installing?

 

[larshenrikoern]

Does the current behavior impact the systems actual performance ???. My wild guess is that waking from sleep is actually a boot from the swap device. In that case a few files will be needed to stay there because you cant delete / (in this case a root mounting point that will be on the swap device).

 

[Tracker]

Does the current behavior impact the systems actual performance ???.

Yes, random crashing of tabs in a heavy browsing session. Sometimes accompanied by slowing down of processes and increased system load (usually during browsing mostly - despite the system having more than enough free memory)

My wild guess is that waking from sleep is actually a boot from the swap device. In that case a few files will be needed to stay there because you cant delete / (in this case a root mounting point that will be on the swap device).

Not sure about this - when the system starts there is usually nothing in swap - so i'm not sure how system dependent files might be in there. What do you mean by "boot from swap device" ? Maybe I am not able to understand.

General question : are there not any sysctl values that I could change and see if there is any difference and helps empty swap? Hopefully without rebooting

 

[larshenrikoern]

Yes, random crashing of tabs in a heavy browsing session. Sometimes accompanied by slowing down of processes and increased system load (usually during browsing mostly - despite the system having more than enough free memory)

Not sure about this - when the system starts there is usually nothing in swap - so i'm not sure how system dependent files might be in there. What do you mean by "boot from swap device" ? Maybe I am not able to understand.

General question : are there not any sysctl values that I could change and see if there is any difference and helps empty swap? Hopefully without rebooting

We are talking about resume from sleep state. Right ???
I found this:
https://freebsdfoundation.org/wp-content/uploads/2018/04/Pain-and-Suffering-on-the-Road-to-Resume.pdf
From what I read there it is actually a boot from the sleep device (ram or swap). So I have a reason to believe that moves the / mountpoint to the boot device used by hibernation to somewhere else than a normal boot.

 

[larshenrikoern]

General question : are there not any sysctl values that I could change and see if there is any difference and helps empty swap? Hopefully without rebooting

A bit old but look here
https://www.reddit.com/r/freebsd/comments/2k6hq0/swap_virtual_machines/

 

[smithi]

We are talking about resume from sleep state. Right ???
I found this:
https://freebsdfoundation.org/wp-co.../Pain-and-Suffering-on-the-Road-to-Resume.pdf

Thanks, an excellent article by John Baldwin, which can be considered authoritative.

Nothing in there indicates any connection between suspend /resume and swapspace use, for S3 state anyway.

From what I read there it is actually a boot from the sleep device (ram or swap). So I have a reason to believe that moves the / mountpoint to the boot device used by hibernation to somewhere else than a normal boot.

FreeBSD never did implement S4 state (suspend to disk, aka hibernation) as jhb@'s article makes clear, excepting a few older (mythical?) systems that offered an 'S4bios'.

S3 resume loads nothing from disk, it's all in RAM.

I don't know if geli does anything about encrypted swap on suspend; I assume its docs would discuss that if so?

 

[larshenrikoern]

Could the reason for Trackers problem be that something happens with the geli keys. If he uses the method of just adding *.eli in fstab it will encrypt with a onetime key. But does that survive the S3 state in a way that it is reinitialized ?? If not the system cannot find the contents of the drive leading to crashes and other random behavior. Just as Tracker described.

 

[grahamperrin]

Please note:

• Sleep resume - caveats and gotchas.

– I'll add some comments there.

… random crashing of tabs in a heavy browsing session.

Which browser?

Packages of ports from quarterly, or latest?

pkg -vv | grep -e url -e enabled -e priority

Can you let us have the address of a crashing tab?

What's your graphics hardware, and driver?

… Sometimes accompanied by slowing down of processes and increased system load (usually during browsing mostly …

Can you describe the system load?

A screenshot of htop, if you like. Thanks.

 

[Tracker]

Guys - I am about to lose some unsaved work in the browser due to this swap issue. I managed to get a pendrive with 8G capacity (as big as swap space)

How do I somehow manage to salvage the situation as described here? by mer

Do you have an external device, even a USB drive around?
If so have you tried creating a swap partition on it, plugging it in and doing a swapon to that just to see if it lets you swapoff the encrypted swap?

The current browser session has become unresponsive - but still showing on the workspace. I am guessing by trying this it can be salvaged.

Please help.

 

[mer]

Assuming you are going to use the whole pendrive, it shows up at da0 when plugged in, all commands done as root or sudo:
first clear any type of partitioning on it
gpart destroy -F da0

second set it up for gpt partitioning. NOTE this does not create any partitions, just the scheme
gpart create -s gpt da0

third create a freebsd-swap partiton over the whole thing NOTE that is a lowercase "L" (for label) before tmpswap
gpart add -t freebsd-swap -l tmpswap da0

fourth just to check NOTE should show 1 partition, type freebsd-swap, label tmpswap over the whole device
gpart show -l da0

Verify that the partition shows up.
ls -ltr /dev/gpt/tmpswap (NOTE this only works if you have not disabled gpt lableling)
ls -ltr /dev/da0* (NOTE should only show up da0p1)

Use it as swap
swapon /dev/gpt/tmpswap
or
swapon /dev/da0p1

Now start moving the original swap
swapoff /dev/ada0p4

 

[Tracker]

gpart destroy -F da0

It's da1 for me - but trying it says "device is busy"

sudo gpart destroy /dev/da1
gpart: Device busy

Tried unplugging the usb and plugging it back in - same result
also gpart show results just for clarity

gpart show da1
=>      64  15646593  da1  GPT  (7.5G)
        64   9613396    1  ms-basic-data  (4.6G)
   9613460     10068    2  efi  (4.9M)
   9623528       600    3  ms-basic-data  (300K)
   9624128      1472       - free -  (736K)
   9625600   6021057    4  linux-data  (2.9G)

 

[mer]

sudo gpart destroy /dev/da1 gpart: Device busy

You missed the "-F" I had in my command. -F means "force". Your error message means "You have partitions on the device, either delete them all or use the -F to force things"