Solved Out of memory with nothing running

J

Jason_25

Let's put down some ground rules first.
1. This system has 64GB RAM
2. Nothing is running just KDE sitting at the desktop
3. Free RAM is less than 2 GB
4. I am not going to be messing with the ZFS ARC cache - besides in testing 2/4/8/16/32GB RAM systems still eventually become unusable after this
5. Low power computers that run for days from the UPS systems I build cannot contain 128 GB RAM so I cannot upgrade (only giant power hungry servers can use 128GB RAM)
6. In fact memory is low enough to cause a huge detriment to performance - Falkon browser hangs with 1 tab open. No need for the "FreeBSD uses all RAM and there is not a problem" kind of gas lighting that is so common. Not gonna work with me, sorry.
7. Rebooting fixes it instantly but I am not doing that just because FreeBSD manages memory worse than Windows 95.
8. All performance was lost after using DD to copy some files
9. All signs point to ARC cache but I am still not tweaking it just to browse a webpage. FreeBSD memory manager is broken.
10. Not the first time I have run out of memory with FreeBSD. This has been a constant theme for the past 15 years. I cannot use Linux on this system because - paradoxically - I require ZFS on all available systems and I can no longer tolerate using Alsa with one sound source at a time. If I tolerated systemd I could install Debian with ZFS and pulseaudio but that is not happening. Obviously not using Windows. The short of it is that all operating systems are bad. So no need to tell me to run somewhere else. There is nowhere else.

Ultimately I am just going to have be on the lookout for newer motherboards that support low power processors and huge RAM amounts. Cost is largely irrelevant. I would pay for a system that kept on running for years without needing to be rebooted or fooling around with caches.
 
Unmounting the USB device that I did the dd copy to completely freed all RAM. Around 55GB RAM went from "Inact" to "Free" immediately upon unmounting.
 
So the system responsiveness increased greatly. That fixed the problem, or at least worked around it. The system goes from not being able to stream 1 video to being able to open as many tabs as I would like.

I think the memory caching should be more dynamic. So if the system senses that a copy operation is taking place it allocates more memory for file caches. It copy operations are not taking place then it starts to free it up. Not perfect because sometimes you would want the memory full before any copying demand is placed on it.
 
ARC in general will expand to use almost all of free memory. This is highly dependent on usage patterns, but if you start to hit ARC limits and need memory, ARC will get freed. But that is not instantaneous which can cause issues. ARC is basically "Read Cache"

Jason_25 said:
I think the memory caching should be more dynamic.
Click to expand...
Define exactly what you mean here. "Memory Caching" of what exactly?
Free vs Inact memory, not sure it actually means anything. Inact is basically "can probably be reclaimed and moved to Free but nothing has triggered that transaction"

Based on Post#1 and #4 I'm not sure what the use case/pattern is going on. #1 says nothing about what you are trying to do but sounds like "sitting idle at a KDE desktop". What is actually running in KDE when sitting idle? Something trying to index everything which touches every file which leads to disk cache usage? Then we jump to #4 and are at "streaming 1 video to as many tabs as I want"

dd from A to B are you really using dd to "copy some files"? That seems odd to me unless you are using dd to put an iso image on a flash device.

I'm not saying you aren't seeing issues, but I'm trying to see any information that "lets us help you".
 
No zpool on the USB device. It happens to use ext4.

I did use DD to copy a huge ISO because I enjoy it's status output compared to cp with ctrl-t or rsync.

I do not have a clue about memory caching. I do not know if that is even the problem. What I do know is that unmounting the drive that I copy to/from speeds the system up from being able to not stream a video at all to being able to again. Unfortunately I am back here to say that not all performance has been regained and I am going to have to restart unless I get any more tips.

This is all happening on my HTPC/media PC. It does archival storage as well as capturing HDMI at 1080p over USB3 from the Fire Stick. It plays streaming videos from Youtube usually at 1080p but can stream 4K very well if necessary. It also plays Bluray from ISO not from the disc because FreeBSD does not have support for Bluray's filesystem - FreeBSD can suprisingly create 1080p and 4K ISO from compatible drives - If you compile VLC from source on FreeBSD with libluray support you can play back 1080p encrypted bluray - 4K you need to compile makemkv or use a Linux PC to decrypt. Unfortunately loading a 90 GB video into memory with VLC is putting huge memory pressure on FreeBSD as well. It has a secondary USB ethernet for backup as well.

I have 5 FreeBSD systems right now btw. This workstation (64GB), the media computer (64GB), a cable gateway(16GB), a virtualization server(32GB) and a cellular gateway/stream server(16GB). All with ZFS. I do not use swap either and my all SSD fleet thanks me for that. I have other internet connections and computers too that is just my BSD stuff which is the majority of it.

My feelings on tweaking caching - ARC - whatever it is - is that we should not have to do it. I have a car with a bad design for a cam position sensor and it requires an arcane and obsolete alignment procedure to change it. Changing it is considered part of the engine rebuild procedure. Every other car it is 2 screws and 5 minutes of time and away you go. Sound familiar?

Again I am going to take a short break to exercise and if no one has any more tips I am just going to reboot the media system.
 
Jason_25 said:
No zpool on the USB device. It happens to use ext4.
Click to expand...

You said the problem went away when you unmounted the filesystem on the USB device, which does not have ZFS on it. So the ARC or other ZFS memory is not your problem.

Having said that, the buffer cache associated with that filesystem is dropped on unmount, so while it is not the arc there might still be a good chunk of memory free'ed there.

But FreeBSD doesn't use nearly as much non-ARC cache as Linux. So something is odd here.

The DD copy, was the target the USB disk?

It would be helpful if you could post the output from top sorted by resident memory `top -o res -d 1`.
 
I wanted to clarify that the problem did not completely go away when the disk was unmounted. If I had to quantify the speed increase it would be about 25%. But in order to get the other 75% back I am going to need to restart most likely.

I do not have much experience with ZFS on Linux. Only on Proxmox when I used it for a while.

The dd target was to a file in the unorthodox way on a disk with an ext4 filesystem and probably from it also and to a disk with a UFS filesystem and probably from it as well. I have been messing with a lot of disks recently so this is all blending together. Maybe using dd in this strange way from file to file instead of file to device or device to file tripped up the kernel?

`top -o res -d 1`:
Code: 
last pid: 48803;  load averages:  0.32,  0.28,  0.50   up 98+14:35:41  00:08:32
54 processes:  1 running, 53 sleeping
CPU:  0.3% user,  1.6% nice,  0.4% system,  0.1% interrupt, 97.5% idle
Mem: 456M Active, 1049M Inact, 45M Laundry, 10G Wired, 48G Free
ARC: 8225M Total, 1533M MFU, 5964M MRU, 512K Anon, 34M Header, 692M Other
     6953M Compressed, 7929M Uncompressed, 1.14:1 Ratio

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
47421 jason        85  20    0   974M   461M select   0  18:27   4.98% falkon
48797 jason        20  40   10  8896M   460M uwait    1   0:07  21.63% QtWebEng
47381 jason        36  20    0  1050M   315M select   2   0:07   0.68% plasmash
47845 jason        12  36   10  8627M   153M uwait    3   0:01   0.00% QtWebEng
47359 jason        12  20    0   476M   143M select   1   0:04   0.00% kwin_x11
47357 jason        16  20    0   413M   137M select   0   0:03   0.00% kded5
48796 jason        14  36   10    16G   136M uwait    4   0:00   0.00% QtWebEng
48799 jason        12  45    0   456M   136M select   2   0:00  23.10% konsole
47325 jason        14  20    0   316M   133M select   3   1:11   0.78% Xorg
47418 jason         9  20    0   452M   131M select   4   0:25   0.00% ksysguar
47426 jason         5  20    0   345M   127M uwait    6   0:16   0.00% QtWebEng
47387 jason         9  20    0   427M   121M select   1   0:02   0.00% Discover
47375 jason         8  20    0   321M   110M select   2   0:00   0.00% ksmserve
47380 jason         8  20    0   320M   110M select   1   0:00   0.00% kaccess
47347 jason         8  20    0   318M   107M select   1   0:00   0.00% klaunche
47377 jason         5  20    0   134M    56M select   7   0:00   0.00% org_kde_
47343 jason         3  68    0   117M    50M select   3   0:00   0.00% plasma_s
47328 jason         3  21    0   117M    49M select   3   0:00   0.00% startpla
47346 jason         1  42    0   113M    48M select   1   0:00   0.00% kdeinit5
47404 jason         7  20    0   131M    42M select   2   0:00   0.00% kactivit
47363 jason         3  20    0    96M    31M select   0   0:00   0.00% kglobala
47389 jason         4  20    0    97M    30M select   2   0:00   0.00% gmenudbu
47379 jason         3  20    0    91M    29M select   4   0:00   0.00% xembedsn
47374 jason         3  20    0    85M    24M select   1   0:00   0.00% kscreen_
47392 jason         9  20    0   381M    11M select   6   0:03   0.00% pulseaud
47367 jason         4  26    0    31M  6140K select   6   0:00   0.00% dconf-se
47420 jason         2  20    0    17M  4608K select   2   0:12   0.00% ksysguar
47372 jason         1  68    0    16M  4068K select   3   0:00   0.00% xsetting
 6926 jason         1  20    0    16M  3900K select   4   0:00   0.00% screen-4
48801 jason         1  36    0    14M  3856K wait     1   0:00   2.98% bash
47339 jason         1  20    0    14M  3752K select   2   0:00   0.00% dbus-dae

The 16G QtWebEngineProcess is from Falkon sitting with one tab open with the Youtube homepage. Usually the system can handle this and many more Youtube videos open after a fresh restart. Now it is tripping up with one video open because I have not restarted yet.

You can see that CPU and indeed memory usage is not a problem yet videos are messing up badly. Almost as if the AMD graphics driver has had it's memory scrambled somehow during all the file copies and 4K ultra resolution video playback and heavy USB device usage.
 
Nothing stands out that would assassinate youtube. Maybe you want to post the top output right after a reboot when youtube works and right before a reboot when it doesn't.

I heard before that ZFS can be a problem for video playing (including on Linux), but I have been unable to reproduce.
 
I did finally "restart" because the battery died while I was playing a video because the charger is not powerful enough, the wiring not thick enough and the battery was 7 years old so it had a lot of self-discharge going on.

Obviously the performance is fine for now after the restart. Also the 4K videos were taking about 2 minutes to start showing and had some weird status for VLC in top indicating a storage hangup of some kind. After the restart they load up immediately. I am backing up the mirrored drives to another external SSD right now so we will see if videos still hang up after this transfer or if this whole incident was more of a fluke.

I also replaced the A/V receiver due to a problem playing back Dolby Digital 5.1 over optical (HDMI from media to TV then optical from TV to receiver - Top Gun Maverick was the problem movie - The 5.1 was glitching on and off) from the media system. The problem was not with FreeBSD on this one though. Surround sound seems to work well on FreeBSD with no extra configuration. I never got it to work with Linux so this takes me back to using a Windows system many years ago With an Nvidia Nforce 2 motherboard with always on Dolby Digital 5.1 Optical SPDIF sound. I am still using optical with the new receiver even though modern systems only play surround sound signals when a movie or capture source actually plays it as the source and even though the new receiver has HDMI and Dolby Atmos built in. Yes, FreeBSD passes the Dolby Digital 5.1 signals from the Fire Stick to the media computer to the TV and then to the receiver so not just Bluray discs have 5.1 audio on this system (media computer just has HDMI out and relies on the HiSense TV with optical out in "Passthrough" mode to send the 5.1 signals correctly). I also have 5.1 and DTS test videos that work perfectly. Go FreeBSD.

As I mentioned I may need to upgrade the power supply/charger for this system if 4K videos drag the voltage down too much near the end of a long movie. It is a homemade laptop thing like all my others.
edit: I did replace the battery with a new one after it died yesterday also.
 
You must log in or register to reply here.
Back
Top