How big should /tmp be?

H

hruodr

I have in /etc/fstab of my pxe client:

tmpfs /tmp tmpfs rw,mode=1777,size=128m 0 0
tmpfs /var/run tmpfs rw,mode=0755,size=20m 0 0

The second only to avoid booting errors due to locking.

But the small /tmp breaks pkg. I had to run:

mkdir /var/tmp/pkgtmp
env TMPDIR=/var/tmp/pkgtmp pkg update -f
env TMPDIR=/var/tmp/pkgtmp pkg ...
 
I generally tmpfs /tmp but the advice is not to do it for /var/tmp (even though its a symlink on some platform).

I originally read this on a NetBSD and Gentoo handbook but its echoed by the ArchLinux wiki:

Do not use it on /var/tmp, because that directory is meant for temporary files that are preserved across reboots.
Click to expand...

As for /tmp, do you need to set a size? At least on OpenBSD (mfs), I can set it to 20G and it only uses memory as needed (or fails if too large).
 
Sounds like an interesting corner case. tmpfs causing in memory filesystem. sounds like the size parameter is the upper limit, so sparse allocation instead of preallocation of the full amount.
What amount of memory do the pxe clients have? it may be a good data point to not set sizes, and see how much is used to get you rough upper limits.
 
I just have "tmpmfs=YES' in /etc/rc.conf and leave it at that. Then again, it only ever has a bunch of small files in that directory, I never write anything large into it, so I haven't stress-tested it. The manpage for mdmfs doesn't state a default size for the '-s' argument if it's not specified.
 
I just tested it. By default the size is 20 megabytes, specifically 20480 1k-blocks, so it's not very large. And it doesn't expand if you write a large file to it, it fills up and you get a no space left error.
 
I tried setting 'tmpmfs=yes' and 'tmpsize=4g', and rebooted. Then df reports 4 gigabytes available in /tmp, however htop shows only 374 MB memory actually used, so it's space-efficient. If I then copy a 3.5 GB file to /tmp, df shows /tmp is almost full, and htop shows 3.6 GB memory used. So it appears that it doesn't allocate the memory from the kernel until you write to it.

Subsequently deleting the file does not result in a corresponding drop in how much top thinks is in use, however. But hopefully the driver is intelligent enough to mark that memory as available for the kerne to grab back at some later time. It would be unfortunate if writing a single large file to /tmp, then subsequently deleting it, results in a large amount of memory permanently allocated to the mfs driver and never released back to the kernel.

So yes, the tmpsize parameter in rc.conf specifies the maximum size it is allowed to grow to, but by default it only uses as much memory as is needed to accomodate the files that are written into /tmp; it allocates memory on demand from the kernel as files are written to it. The default maximum size that you get if you don't specify 'tmpsize' is 20 MB. At least, that's what I observe on my 14.5R system here this evening.
 
mer said:
Sounds like an interesting corner case. tmpfs causing in memory filesystem. sounds like the size parameter is the upper limit, so sparse allocation instead of preallocation of the full amount.
What amount of memory do the pxe clients have? it may be a good data point to not set sizes, and see how much is used to get you rough upper limits.
Click to expand...

This filesystem has its file contents paged to swapspace. So you can make it quite big without taking too much RAM, assuming you have sufficient swapspace.
 
I've had no problems doing pkg upgrades on my box with the default /tmp size; but I'm running UFS on what is now quite a small laptop, and only have a thin window manager, not a big desktop environment, probably the largest piece of software I've got installed is libreoffice. For the time being I've set my tmpsize to '256m' in rc.conf, to allow for some headroom, 20M does seem a bit on the low side nowadays. I'm not aware of any 'standard' installation, there are so many variables to consider. It would be interesting to know what /tmp size ghostbsd uses to build a desktop, though.
 
hruodr said:
Have some a "standard" installation with /tmp as tmpfs? How big?
Click to expand...
I think it depends a bit on the server’s role.
For example, I have servers with a very large /tmp because the server stores large files there for deploying them over the network.

In general, I set the size of /tmp to 4 GB on typical servers.
 
blackbird9 said:
I tried setting 'tmpmfs=yes' and 'tmpsize=4g', and rebooted. Then df reports 4 gigabytes available in /tmp, however htop shows only 374 MB memory actually used, so it's space-efficient. If I then copy a 3.5 GB file to /tmp, df shows /tmp is almost full, and htop shows 3.6 GB memory used. So it appears that it doesn't allocate the memory from the kernel until you write to it.

Subsequently deleting the file does not result in a corresponding drop in how much top thinks is in use, however. But hopefully the driver is intelligent enough to mark that memory as available for the kerne to grab back at some later time. It would be unfortunate if writing a single large file to /tmp, then subsequently deleting it, results in a large amount of memory permanently allocated to the mfs driver and never released back to the kernel.
Click to expand...

I tried repeating that and the space was freed immediately. The point of tmpfs is that it's its own filesystem, so it knows when a file is deleted; the older md devices use UFS and wont free memory unless the deleted file is written to swap. Do check that you are actually using tmpfs, mdmfs can fall back to an md file if tmpfs creation fails, but that's not likely - probably some software has the file open.

The default maximum size that you get if you don't specify 'tmpsize' is 20 MB.
Click to expand...

That comes from the rc defaults. It doesn't apply if you use fstab or create the tmpfs manually.
 
As others have mentioned, YMMV. The size of /tmp tmpfs is not cut and dry and will be determined by the purpose of your server/workstation. For my own use cases I have it at 4GB since I use it as a cache location for media transcoding and I want fast IO access.
 
kent_dorfman766 said:
As others have mentioned, YMMV. The size of /tmp tmpfs is not cut and dry and will be determined by the purpose of your server/workstation. For my own use cases I have it at 4GB since I use it as a cache location for media transcoding and I want fast IO access.
Click to expand...
That's rarely as effective as people think:

tmpfs
$ dd if=/dev/random of=/tmp/deleteme bs=1m count=4096 ;rm /tmp/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.549101 secs (449777126 bytes/sec)

HDD
$ dd if=/dev/random of=~/deleteme bs=1m count=4096 ;rm ~/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.419404 secs (455970160 bytes/sec)
 
bob2112 said:
I tried repeating that and the space was freed immediately. The point of tmpfs is that it's its own filesystem, so it knows when a file is deleted; the older md devices use UFS and wont free memory unless the deleted file is written to swap. Do check that you are actually using tmpfs, mdmfs can fall back to an md file if tmpfs creation fails, but that's not likely - probably some software has the file open.
Click to expand...
Interesting, I am using a UFS filesystem so perhaps an md device was created. It might be more effective to define a tmpfs filesystem in /etc/fstab for /tmp, rather than use the /etc/rc.conf configuration directive.
 
I just checked my system, and a tmpfs filesystem was created by the rc.conf directive, after all.

# mount | grep tmp
tmpfs on /tmp (tmpfs, local)

So using the rc.conf directive works fine, it creates a tmpfs device.

The description here https://www.baeldung.com/linux/tmpfs-size-dynamics relates to the linux version of tmpfs, but I would have expected freebsd's version to behave similarly.

Quote:-

"5.1. Memory Prioritization Mechanics​

Linux treats tmpfs memory similarly to the memory used by applications.

When system memory becomes scarce, the kernel’s memory manager decides, based on usage patterns, which data to keep in RAM and which to offload to swap space. Here, frequently accessed data (“hot” data) is held in RAM for fast access, while less frequently accessed data (“cold” data) may be moved to swap space.

Moreover, this prioritization is evident when tmpfs and applications concurrently demand more memory than is physically available.

For example, suppose tmpfs is consuming a significant portion of memory (say, 40% of physical RAM) and running applications suddenly require more memory (e.g., 70% of physical RAM). In that case, the kernel will decide what to keep in RAM and what to swap out based on the access patterns and importance of the data."

So the kernel memory manager decides on the fly how memory is allocated to the tmpfs device, when it is released back to the kernel, moved out to swap space, etc. Hence the result will depend on what else is running on the machine, how much free memory it has, what the current load is, etc. So the observed result may not always be the same, depending on the current runtime situation of a particular machine. Which probably explains why we had inconsistent results in this instance.
 
bob2112 said:
That's rarely as effective as people think:

tmpfs
$ dd if=/dev/random of=/tmp/deleteme bs=1m count=4096 ;rm /tmp/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.549101 secs (449777126 bytes/sec)

HDD
$ dd if=/dev/random of=~/deleteme bs=1m count=4096 ;rm ~/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.419404 secs (455970160 bytes/sec)
Click to expand...
In both cases dd is going to write to the kernel's filesystem buffer cache in the first instance, which is memory. If the kernel has a lot of free memory available, it is able to use that for filesystem write buffer cache, so you are effectively writing to memory, and using a tmpfs target provides no advantage over the HDD.

What happens if you pass an additional 'oflag=direct' argument to dd, to (hopefully) bypass the kernel filesystem buffer?
And if you specify synchronous writes?

From the dd manpage:-

oflag=value[,value ...]
Where value is one of the symbols from the following list.
fsync Set the O_FSYNC flag on the output file to make writes
synchronous.

sync Set the O_SYNC flag on the output file to make writes
synchronous. This is synonymous with the fsync value.

direct Set the O_DIRECT flag on the output file to make writes
bypass any local caching.

You can also try adding 'conv=fsync' to the dd command to force dd to call fsync(2) to flush the buffer cache.

So for example try
dd if=/dev/random of=~/deleteme bs=1m count=4096 oflag=sync,direct conv=fsync

and then see if there is any difference in execution time between writing to the tmpfs and HDD targets.

We also need to consider that your HDD itself probably has an onboard hardware RAM cache, which may be quite large, depending on the drive model. On linux a drive's write back cache can be disabled using 'hdparm -W 0'. Whereas it appears freebsd has sysctl interfaces to disable drive write caches https://forums.freebsd.org/threads/is-there-a-simple-way-to-disable-write-cache.56367/

So you actually have to work quite hard to bypass all the various type of caching in the system to get to the point where you see a difference in write speed due to the underlying mass storage device itself. Of course how soon you reach the point at which you do observe a difference will depend on how much free memory the kernel has available to allocate to the filesystem buffer, and the size of the data being written. The speed difference is more apparent when you write directly to a device, as when dd'ing directly to a usb flash drive; the filesystem cache is not used in that case, since you are not writing to a filesystem..
 
bob2112 said:
That's rarely as effective as people think:

tmpfs
$ dd if=/dev/random of=/tmp/deleteme bs=1m count=4096 ;rm /tmp/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.549101 secs (449777126 bytes/sec)

HDD
$ dd if=/dev/random of=~/deleteme bs=1m count=4096 ;rm ~/deleteme
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 9.419404 secs (455970160 bytes/sec)
Click to expand...

Your example non-comprehensive.
1) you only benchmark a specific data write situation
2) you don't take into account the effect of different conv= and oflag= options to limit or disable buffering
3) it is obvious that your target hard disk is an SSD device, which is essentially solid state memory, so may have access times similar to a RAM filesystem

transcoding using a tmpfs cache is (generally) going to be a more efficient process, especially if the transcoding process is not purely sequential IO operations.

Yeah, what blackbird9 said, but I didn't see his response before writing my own.
 
blackbird9 said:
In both cases dd is going to write to the kernel's filesystem buffer cache in the first instance, which is memory. If the kernel has a lot of free memory available, it is able to use that for filesystem write buffer cache, so you are effectively writing to memory, and using a tmpfs target provides no advantage over the HDD.
Click to expand...
That's my point.

kent_dorfman766 said:
3) it is obvious that your target hard disk is an SSD device, which is essentially solid state memory, so may have access times similar to a RAM filesystem
Click to expand...

It's on NAS drives, but the speed of the backing store doesn't become relevant unless you fill the write buffers or run out of disk cache; that's why I chose a 4GB file, the maximum size of your tmpfs. If you don't have the ram for buffering & caching, the chances are that you don't have enough for tmpfs.

I don't know what you are using cache for exactly, but if if you are transcoding to and/or from /tmp, the difference is likely to be dominated by the extra copying.
 
You must log in or register to reply here.
Back
Top