Solved How much space for /var?

  • Thread starter Thread starter Deleted member 43773
  • Start date Start date
D

Deleted member 43773

Guest
Background:
I'm thinking of moving /var into a RAM-disk.
(" FreeBSD and Solid State Devices " explains how it may work and why it could be not a bad idea)
For me it seems it may gains a small bit additionally performance - if one have enough RAM (64G) it's at least worth a try.

However, for Traditional Split File System Partitions HB 2.6.3 recommends 2G for /var.
On my systems so far I have no own partition neither own drives nor quotas for /var.
#du -hs /var shows me 5.4G, 15G ... anyhow significantly more than 2G on my systems.

Here my question:
What happens if /var runs out of space?

On a 2G partition this already would have been the case on my systems.
Will the system just delete old stuff or move it elsewhere (e.g. some kind of /usr/.../archive/old/garbage/...)?

Or to put it another way:
I know the system is not actually writing all data immediately to disk but buffers it and writes it if it's a good opportunity (there are buffers to be synchronized as far as I understood), and also the amount of data for /var is not really large,
but is there any possibility or tool to measure the actual average read/write-load over a couple of hours, a day or two, or do I need to run a script via cron counting bytes?
For not to be misunderstood:
I don't want to make a benchmark test, measuring the possible maximum data transfer speed but the actual data load on a fs/drive while the system works.
 
it depends
you can use symlinks or if zfs is available create datasets for /var/db /var/whatever uses the space
if you use zfs you can see dataset statistics at
sysctl kstat.zfs.ntank.dataset
 
...ah, yes, right... ZFS gives a lot of information - I'll check that out!
Thanks!
 
You can also use the du command to measure current usage.

Personally, I would not move /var/log to a RAM disk. For debugging problems, it is super helpful to have a track record of how the system has behaved. Maybe keep a copy of /var/log on a persistent disk, and update it regularly (every 30 seconds or so).

To your question: What happens when /var runs out of space? Fundamentally, expect the system to come down like a house of cards. For example, processes that try to write to /var/db or put PID files into /var/run will all crash. It will get very ugly.
 
Thanks. I will check that out also.

I understand why the system will crash horribly when /var is not working anymore, because it's not only logfiles saved there but all running processes having data there (that's why I think it could increase performance).
But then the HB's recommondation with 2G would make no sense since my systems gained way over 5G within a couple of years...
So either there need to be some system's mechanism to keep /var running, like preventing /var to flow over, such as deleting old stuff, moving it to somewhere else [there are lots of old logfiles on /var not really in current use, which could be archived somewhere else], or most systems that actually have a 2G /var would not run anymore after 2..3 years...
 
Just as one real-world example: I use separate file systems for /var and /var/log, so running out of space in /var/log doesn't immediately kill the system (not that it has ever happened). My server has about a dozen services running, but no windowing/GUI/DE.

Space usage in /var/log is 267 MiB. That is dominated by two home-brew services I run (a backup system and a home equipment monitoring/control system), which are both "works in progress", and log extensively so I can debug when things go wrong (as they invariably do). They don't use logrotate to clean up their logs, and I only do it manually when I feel inspired. I bet I could cut that down to half or a third if I cleaned up the logs, but I'm too lazy for that.

Space usage in the rest of /var is 1052 MiB (tiny bit more than 1 GiB). That is dominated by the freebsd-update and "pkg update" caches and rollback information. Cleaning those up reduced it to 307 MiB, for a total in both parts of /var of 574 MiB.
 
Well, that's all significantly below 2G. I also figured out: logs are not that much, a couple of MB. Of course it's pure text only.... Maybe I need to figure out first, what fills up my /var to above 15G
 
Profighost said:
Maybe I need to figure out first, what fills up my /var to above 15G
Click to expand...
Drill down: du -sk /var/* | sort -n. Look where the biggest usage happens, then do the same for that directory; du -sk /var/somedir/* | sort -n. Keep doing that until you find where most of the data is.
 
As you put your primary question in bold I'm assuming that's the main point you want to have answer to - what happens when /var is full. Well you can always spawn a VM and simulate it.
You have /var/cache/pkg which can get big very fast.

What does it mean "ram-disk"? Do you mean tmpfs ? /var/db is important enough to have on permanent storage though. How would you deal with it (symbolic link to a permanent storage?). I don't think you'll gain much performance out of this.

To monitor /var you could:
a) create cron script and enumerate the actual size of /var each cycle
b) use find /var -type f -mtime -1 daily to see what changed
c) use dtrace to monitor the usage
 
Thanks SirDice, I'll check that out!

Code: 
.
..
...
869807        /var/crash
3383440     /var/db
11682759    /var/cache

1           /var/cache/cups
11682758    /var/cache/pkg

pkg seems to be the suspect...


_martin:
Exactly. Thanks for somebody remembering my main question :)
More precisely:
Is there any system's internal mechanism to prevent /var from overflowing thus killing the system?
I mean, as I already told: 2G "traditional standard" and my machines already have way more... Does the system fills up /var to a certain point and then starts to delete old stuff? I cannot imagine, that over time the system lets /var uncontrollable flow full until space expired and just crashes... with 5...15G on my machines within 2...3 years there should be many systems with 2G /var that would have been crashed...

However, my idea of putting /var into a RAM disk isn't really that matured yet - that's why I'd like to get some input by you experienced guys.
I thought, the RAM disk would be initiated at systemstart, and saved back, when the system shuts down.
However, if some of you'd say: "there will be no signifcant performance gain until you're on a ARM with slow Flash-Mem as disk; you'll run into more probs, than gaining much benefit..."- I'd have that as some kind of experiment for any time later.
 
Profighost said:
Is there any system's internal mechanism to prevent /var from overflowing thus killing the system?
Click to expand...
For /var/log, yes. That's what newsyslog(8) does. Everything else in /var, no.

Profighost said:
I mean, as I already told: 2G "traditional standard"
Click to expand...
It's just a ball-park figure for a 'standard' machine that doesn't do much. It's not a "hard" figure because nobody knows what you do on your machine.

Other than that, it's your job as a system's administrator to keep an eye on this. Expand the filesystem if necessary or split off /var/db/mysql for example (if you have mysql installed) to it's own filesystem.
 
Do you use packages? If yes, have you ever used pkg-clean(8)? If you've never used pkg-clean(8) but did install or update lot of packages over the years /var/cache/pkg can collect a lot of old, cached, packages. Definitely clean those on a regular basis. I have my own repositories, I have no real need for those cached packages. So I always run pkg clean -a after I'm done installing/updating packages.
 
So, bottom line:
It simply would be easiest and best just to let /var stay within standard / partition because I always have an eye on overall disk space
- unless I am really know what I am doing including to know the growth of my /var.
 
SirDice said:
Do you use packages? If yes, have you ever used pkg-clean(8)?
Click to expand...
Yes. From many sides it seems to be recommended to prefer pkg installation instead of ports. And because to prevent to mix both up and pkg is way faster than make a port I install by ports only for the last 2...3 years.

pkg clean? I only use auto ... OH-HA!

Code: 
.
..
...
    /var/cache/pkg/libgd-2.3.1,1~8c7c5bc1ec.txz
        /var/cache/pkg/tesseract-4.1.1_2-b24133f487.txz
        /var/cache/pkg/py38-evdev-1.4.0~fc35c346e3.pkg
        /var/cache/pkg/ftgl-2.1.3.r5_6,1-ebae44cba0.txz
        /var/cache/pkg/vlc-3.0.16_7,4~529404e9d7.pkg
        /var/cache/pkg/liborcus-0.16.1_1.txz
        /var/cache/pkg/libXcomposite-0.4.5,1-32bddcd0dc.txz
        /var/cache/pkg/libplacebo-3.104.0~2fa57fdee6.txz
        /var/cache/pkg/webrtc-audio-processing0-0.3.1_2~fe43eaf64a.txz
        /var/cache/pkg/pkgconf-1.7.3,1.txz
        /var/cache/pkg/gstreamer1-plugins-1.16.2.txz
        /var/cache/pkg/fusefs-libs-2.9.9_2~5c42534c47.pkg
        /var/cache/pkg/gtk2-2.24.33.pkg
        /var/cache/pkg/poppler-utils-21.12.0~3ce1c609d4.pkg
        /var/cache/pkg/a2pdf-1.13_2.txz
        /var/cache/pkg/libpsl-0.21.1_1.txz
        /var/cache/pkg/suitesparse-config-5.10.1~210e160d4a.pkg
        /var/cache/pkg/font-winitzki-cyrillic-1.0.3_4~3edb77afc6.pkg
        /var/cache/pkg/bash-5.0.18_3.txz
        /var/cache/pkg/clamav-0.102.4,1.txz
        /var/cache/pkg/fusefs-ntfs-2017.3.23-8d45302feb.txz
        /var/cache/pkg/libdvdnav-6.1.1~b2b5b79e07.pkg
        /var/cache/pkg/font-bh-75dpi-1.0.3_4-2898bcc4f7.txz
        /var/cache/pkg/libdvdcss-1.4.2.txz
        /var/cache/pkg/libXfont-1.5.4_2,2.txz
        /var/cache/pkg/hwloc2-2.4.1.pkg
        /var/cache/pkg/opencv-4.5.3_4.pkg
        /var/cache/pkg/libXinerama-1.1.4_2,1-cdc6997337.txz
        /var/cache/pkg/ImageMagick6-6.9.11.6_2,1.txz
        /var/cache/pkg/glib-2.56.3_8,1-0ff324b255.txz
        /var/cache/pkg/openjdk11-11.0.12+7.1.pkg
        /var/cache/pkg/lynx-2.8.9.1_1,1.pkg
        /var/cache/pkg/lzo2-2.10_1.txz
        /var/cache/pkg/libzmf-0.0.2_22.txz
        /var/cache/pkg/thunderbird-78.6.0~a4750077fa.txz
        /var/cache/pkg/poppler-glib-20.09.0.txz
        /var/cache/pkg/opencv-core-3.4.1_37~cdf2c7e4e5.txz
        /var/cache/pkg/kicad-library-footprints-5.1.10~c90e69dd00.pkg
        /var/cache/pkg/qt5-concurrent-5.15.0~39bb43d978.txz
        /var/cache/pkg/gcc9-9.3.0_1-d8fa04f56e.txz
        /var/cache/pkg/libdrm-2.4.103,1~cbe354f16d.txz
        /var/cache/pkg/bash-5.0.17.txz
        /var/cache/pkg/libdvdread-6.1.0.txz
        /var/cache/pkg/sqlite3-3.33.0,1.txz
The cleanup will free 11 GiB

Proceed with cleaning the cache? [y/N]:

...I think the problem is solved ?
 
System doesn't go down if /var is full. That would be one wtf system if it does. Could you possible have issues? Of course, even headaches. But system doesn't go down because of that. You most likely lose some logs. You may fail to install new packages via pkg as there's no place to cache them. If you're unlucky you could corrupt pkg database (=headaches). But no, system will stay up.

That being said corrupted linker hints files could give you more advanced issues and bigger headaches.
 
ralphbsz said:
Personally, I would not move /var/log to a RAM disk. For debugging problems, it is super helpful to have a track record of how the system has behaved.
Click to expand...
NanoBSD moves /var to a MD along with /etc.
Code: 
~ # mount
/dev/mmcsd0s1a on / (ufs, local, read-only)
devfs on /dev (devfs, local)
/dev/md0 on /etc (ufs, local, soft-updates)
/dev/md1 on /var (ufs, local, soft-updates)
I agree logs are good. For an appliance the wear and tear of /var makes it special.
 
Thanks _martin!
That answers my main question.

I think, all my question are answered satisfatory.
Thank you all for any answer, input, ideas and remarks!
Again, I've learned a lot.
Thanks to all participants.
 
_martin said:
System doesn't go down if /var is full.
Click to expand...
It's complicated.

The kernel won't go down. But services won't be able to start or stop, for example because they can't create or modify /var/run/xxx.pid. Similarly, some services put their sockets and temporary files in various places in /var, and those services will crash or not work. Mail will be rejected (assuming the computer uses mail systems, which today is not all that common); if it's coming from SMTP, it will get spooled, but if it's internal, it will get lost. You mentioned the loader hints; fortunately, I don't think those are updated very often, but if they need to be, some program won't start. And because there is no logging, debugging what exactly got damaged is hard.

So in a nutshell, the bare minimum (kernel, getty, root shell) will work. The admin will be able to log in from the console, and clean up and get things back together. I suspect that logging into a user account with sshd will even work (hopefully, sshd is forgiving enough to allow logins, even when it can't write audit logs records, and can't update /var/run/utx...), but I'm not sure. But the system will not be serving most of its functions, so it's not really "up". It's sort of in an intermediate state.

By the way, similar things tend to happen when /tmp runs out of space: Lots of services will crash. It's not a good place to be.
 
You must log in or register to reply here.
Back
Top