Auto (ZFS) Install - why is compression enabled by default?

ARC is basically read cache, so anything read from a zpool winds up in ARC for some period of time (yes, this is simplified and ignores anything related to tuning). By default ARC may grow to use a lot of RAM (but will get evicted under system memory pressure), so if you take that into account anything does a read may grow ARC, post #25 by Crivens has a good example of something that may grow ARC. Logging may be constantly going on in the background.
If your system is truly idle: no users, maybe just a console logged in running top, not connected to a network absolutely nothing that would cause zpool access (read or write), I would expect ARC usage to maybe slowly climb and then level off.

On all my systems running ZFS I always install the zfs-stats package; it makes it easy to look at different things.
The two I find most useful:
zfs-stats -A gives you ARC summary
zfs-stats -E gives you the efficiency of ARC usage
 
SirDice said:
You sure about that? It's rather interesting that you think an 'idle' system doesn't do anything.
Click to expand...
It's just a fresh install, nope not sure, again I haven't looked into it, perhaps when I have time I may do that. I would have an expectation that idle, is generally idle out of the box, I'm happy with 100mb footprint. Again arc increased along with general memory footprint without releasing over a long period of time. As someone suggested perhaps a cron job and a database etc...

Idle system doing nothing is a generalization. I guess the expectation would be from the perspective of the initial memory footprint, or any process that runs and takes up memory, arc or otherwise temporarily, would release or flush itself at some point. Generally this would be considered a memory leak or in grammar a run on sentence :)

I'm guessing at best , and the guess would fine tuning zfs for out of the box. Will have to install UFS and see if the same thing happens, for simplicity sake. Again I don't have a lot of time to play around, that's why I'm asking questions - thanks all for the help!

Pipeline of ZFS is becoming a lot more clear now! along with benefits of compressing on disk, keeping smaller blocks compressed to memory, and the uncompressed when needing a block. Bootup files may not require that pipeline or perhaps a higher compression for smaller blocks in arc.

Also compression is limited to a thread at a time given that it's linear, albeit very fast with toadys CPU's, compression does not scale well in that sense- it is still a limitiation unless the algorithm has been specifically designed internally to use multiple threads. To my knowledge, none of those algorithms have that ability ... 100 cores, X amount of threads are only limited, no matter how fast to uncompress and decompress one block synchronously of the same block of data ... This can be done asychronously, without compression, and increase that limitation perhaps exponentially. My opinion.
 
Codemonger said:
I've watched my memory grow from a 100mb to 160 mb sitting idle. Nothing is loaded, fresh install , no debug ... I haven't looked into it, but I've also watched arc grow as well.
Click to expand...
To make ZFS perform reasonably well, it caches aggressively. (Unused memory is “wasted” memory, by one way of thinking.)

Disabling compression will only make it need more memory to achieve the same level of caching.

LZ4 compression is absurdly fast compared to gzip/bzip algorithms you may be more familiar with.

The only time I’ve seen any benefit from disabling compression on ZFS was on a dataset used for broadcast video capture with multiple concurrent streams. ZFS would end up not compressing (the MPEG streams aren’t good candidates for further compression, unsurprisingly) anyway so it was just wasted cycles trying (and then aborting) compression, and given the time sensitive nature with a continual stream of new data to save, this ended up being noticeable.
 
Eric A. Borisch said:
To make ZFS perform reasonably well, it caches aggressively. (Unused memory is “wasted” memory, by one way of thinking.)

Disabling compression will only make it need more memory to achieve the same level of caching.

LZ4 compression is absurdly fast compared to gzip/bzip algorithms you may be more familiar with.

The only time I’ve seen any benefit from disabling compression on ZFS was on a dataset used for broadcast video capture with multiple concurrent streams. ZFS would end up not compressing (the MPEG streams aren’t good candidates for further compression, unsurprisingly) anyway so it was just wasted cycles trying (and then aborting) compression, and given the time sensitive nature with a continual stream of new data to save, this ended up being noticeable.
Click to expand...
Yes, the ARC seems to cache very aggressive and manage that memory very well - It will free memory from the arc as required to from what I understand. I think I'm ok with the ARC using memory as required, file system cache in memory is a pretty good use of memory imo - especially treated seperate. I'm trying to figure out why regular memory is being sucked up in conjunction with ARC after the install and not release. Will figure it out.
 
ARC releases its contents either when there is not enough room in ARC for something it sees as a better fit, or when there is not enough room in RAM for other outside requests. The cache does not release just because an amount of time has passed. I believe that is still the case for UFS's caching too. ARC is not separate from the other categories above it and still registers as part of wired if I recall. UFS's cache registers outside those. If the memory is not needed elsewhere, the file cache is kept around until that changes which is where the 'free memory is wasted memory' is referenced to; cache is a tool to do something while free memory waiting to become a tool. It is the same as closing then reopening a program happens faster because the previous program is in memory if the memory wasn't needed elsewhere so is faster than getting it from disk.
 
Mirror176 said:
ARC releases its contents either when there is not enough room in ARC for something it sees as a better fit, or when there is not enough room in RAM for other outside requests. The cache does not release just because an amount of time has passed. I believe that is still the case for UFS's caching too. ARC is not separate from the other categories above it and still registers as part of wired if I recall. UFS's cache registers outside those. If the memory is not needed elsewhere, the file cache is kept around until that changes which is where the 'free memory is wasted memory' is referenced to; cache is a tool to do something while free memory waiting to become a tool. It is the same as closing then reopening a program happens faster because the previous program is in memory if the memory wasn't needed elsewhere so is faster than getting it from disk.
Click to expand...
I thought ARC was separate from wired or perhaps I was reading top wrong, I'm guessing ZFS is tightly integrated into FreeBSD say compared to linux distro's that have ZFS ... Do you see any reason boot-up files would need to be cached and or go through that process? I'll have a look sometime at that - I'm guessing my extra memory addition came from the cron job that someone previously pointed out for the locate.db ; maybe I'll take a look later tonight.
 
Wired is not ARC, but ARC does end up in there. Wired is supposed to be memory that doesn't get reclaimed or swapped out but as memory pressure occurs, an arc pruning kernel thread gets fired up to shrink it if possible (and has been working particularly poorly on 13.3).

FreeBSD is using OpenZFS which is now (13.0+?) based on the ZFS on Linux branch of work instead of the Illumos work due to more development, faster integration of features, bugfixes, etc. Each OS using OpenZFS has its own 'integration' code and can vary in terms of features and bugs according to its implementation differences. I still wish ZFS had more of the integration work that UFS has in FreeBSD but can't see that ever happening...though I'll still dream that one day ARC starts showing in Buf category and gets some of the neat UFS integration tricks.

Reasons for boot up files to be cached: the OS doesn't know what files will be accessed again/next/regularly. To use a file, its contents get read into memory. Why take the time+effort to remove that data from RAM now when it 'might' get used again (and have to be reread to do so) when you can wait until you need that memory to be freed for another process to use it. For those two files as a probable "won't be read again" example, maybe I ran those two through a checksum to see if recmpiling made any change or as a security check to watch for tampering. Your use may vary from mine, which may vary from a web host, which may vary from a kernel developer but if it isn't recently used, frequently used, and there isn't enough free RAM to keep it around then it gets thrown out anyways.

Many files may be used only once per boot, but what if you relaunch that RC script to stop/start a service, or log back out to a login prompt after login, or resync the clock to the network at another time before the next reboot, maybe a device that was present during boot and setup accordingly is later disconnected and after is reconnected. Other than maybe the kernel and init (do these get cached?), what can you tell me is going to be a one time read per boot? Remember, "any" mistake in this answer creates two unnecessary steps of releasing it from RAM (relatively very light load/wait) and reading it from disk again (relatively heavy load/wait). If a mistake is made, now do we cache the file for future use or still drop it immediately again? If you think you have successfully found a correct list of read-once files then you could experiment with making sure they are all on datasets isolated from any multiple read files. You can then `zfs set primarycache=metadata` to exclude the file's contents from arc or `zfs set primarycache=none` to exclude the file's metadata too (doubt metadata was a consideration in this discussion, but in case thought leads down that route...). If I recall, ZFS has a limit of a little over 70 quintillion datasets; if you have too many files scattered in too many places that you want to apply this to, consider symlinking them to fewer common datasets to stay within ZFS's limits...working with so many datasets will also likely have performance impacts on dataset managing/viewing tools.

A more serious time to not cache data is when the data is cached; databases use their own cache, filesystems in virtual machines use their own cache, etc. Caching memory multiple times means more RAM is used and those two copies could lead to them fighting each other and diminishing the value either one has if there is not enough RAM for both at the same time. Some less concerning reasons would be if in a VM where current allocated memory contents need to be written to disk (speed for magnetic or durability for common solid state) and if keeping 1s vs 0s has any noticeable difference in electrical consumption for the device.
 
I thought I included the #s but with 8812MB/s write and 9058MB/s read listed in lzbench, it gives about 15 times slower writing and 2.4x slower reading data from LZ4 compressed RAM than just native RAM use on this old CPU which matters when thinking about ARC instead of disks. For ZFS, there is a small amount of memory that maintains decompressed copies of data when ARC is using compression. Once again that's not the same LZ4 version so numbers will be different, but gives an idea of what could be happening.
 
So I did a fresh install - same parameters with UFS - memory started at 60mb and actually went down to 59mb over time and back up to 60mb. It this state, I found the OS to be what I expect as generally idle. Did not realize arc was part of wired. In comparison to ZFS, where memory reached 160mb from 100mb initial start.

I suspect some fine tuning could go into ZFS on root to mitigate these things ... I see some examples online where people have attempted this. I think some cache hit ratio stats from ARC could tell if that memory is being used effectively perhaps or if its just initial bloat - besides real world use cases.

Thanks all for the answers, it shed some light on the problem I was experiencing.
 
Okay, what you percive as a problem, are you sure it is one? People have explained things. Did you, for example, use a stopwatch to find out how long it takes the two instances to boot up? To do some basic things, like buildkernel? Then we could discuss matters over some facts.
 
Crivens said:
Okay, what you percive as a problem, are you sure it is one? People have explained things. Did you, for example, use a stopwatch to find out how long it takes the two instances to boot up? To do some basic things, like buildkernel? Then we could discuss matters over some facts.
Click to expand...
Yes, I do perceive that as problematic. I'm now really impressed with the UFS footprint along with the idle expectation - awesome work btw :)

ZFS is so powerful and has so many tuneable parameters I'm sure if I spent the time, I could match the UFS footprint as well as sitting idle comparison I'm looking for.

If I go down the road you are suggesting, it would come down to arc cache hit ratio, if arc is effective at not having to load the same file from disk multiple times and ZFS can pull it from cache. I'm not invested in that at this time. Perahaps someone else would be interested in looking at that, would be interesting to check arc statistics out of the box - and see what it says - isn't their a seperate package for that?

Thanks for input / suggestions btw!
 
Off and out to enjoy some sunshine! It looks like primary and secondary cache (arc and l2arc) can be toggled on or off or metadata only for datasets -- that along with compression on / off should be enough to fix some directories --- Perhaps someone out in the virtual realm of the internet - some of the younger geeks - would have some time to do that and make ZFS comparable to UFS on boot and idle.

Thanks all!
 
cracauer@ said:
My hunch is that ZFS itself is heavyweight and if the machine has difficulty coping then compression or not doesn't make much more of a difference.
Click to expand...
It's definitely heavyweight that's for sure, a powerhouse of a filesystem ... Just comparing Apples to Apples - UFS boot / memory footprint / idle no growth vs ZFS boot / memory footprint / idle with memory growth over short period;

Not sure which one boots faster ... ZFS has that seperate stats package that could answer a lot of questions for the above I assume - zfs-stats.
 
Again, unused RAM is wasted (could have the cache of any previously read data, which might prove useful) RAM. Why do you want your filesystem to be so svelte and not make use of the RAM you’ve got? (Especially if it gets out of the way when the RAM is needed for something else, but that’s a somewhat separate discussion.)

There are tons of other benefits from ZFS (boot environments, snapshots, checksums, send/recv) that (for me, at least) make it a no-brainer.
 
Eric A. Borisch said:
Again, unused RAM is wasted (could have the cache of any previously read data, which might prove useful) RAM. Why do you want your filesystem to be so svelte and not make use of the RAM you’ve got? (Especially if it gets out of the way when the RAM is needed for something else, but that’s a somewhat separate discussion.)

There are tons of other benefits from ZFS (boot environments, snapshots, checksums, send/recv) that (for me, at least) make it a no-brainer.
Click to expand...
It's definitely is a no brainer in terms of wanting to use ZFS. 60MB to 160MB of wired ram, I would not consider negligable - relative from one to the other ... ZFS is now available on a bunch of OS's, I'm specifically refering to FreeBSD use of ZFS though not ZFS.

Comparisons are rarely done in regards to top limit, which change drastically over time as well. I have to be honest, I'm not sure why it would bother anyone. Checking on my idle install -- since I fired it up this morning while having a cup of coffee. It would appear that 61MB is now being used. Not much of a change, relative to 60mb.

Also I can see some potential future military applications and not how much unicorn anime porn can be dished out - different strokes for different folks I guess they say, where some things may be more important than others. Keep up the good work freebsd'ers :)

Will look forward to tomorrows cup of Coffee :)
 
true - ZFS makes *heavy* use of memory to improve its performance by various methods of caching. but why not if it is available (and cheap)?
However, if 160MB of used memory are already concerningly much on your system you shouldn't use ZFS at all - and maybe even switch to some (heavily) trimmed-down custom built kernel/OS or a minimalistic variant/build of NetBSD...
 
Codemonger said:
It's definitely heavyweight that's for sure, a powerhouse of a filesystem ... Just comparing Apples to Apples - UFS boot / memory footprint / idle no growth vs ZFS boot / memory footprint / idle with memory growth over short period;

Not sure which one boots faster ... ZFS has that seperate stats package that could answer a lot of questions for the above I assume - zfs-stats.
Click to expand...

With compression on ZFS will be a lot faster for compressible things like the base OS.
 
sko said:
true - ZFS makes *heavy* use of memory to improve its performance by various methods of caching. but why not if it is available (and cheap)?
However, if 160MB of used memory are already concerningly much on your system you shouldn't use ZFS at all - and maybe even switch to some (heavily) trimmed-down custom built kernel/OS or a minimalistic variant/build of NetBSD...
Click to expand...
I never mentioned my system specs, I would expect the same results regardless of whatever system it's on.
VM increase won't change the 160mb .

Why FreeBSD? I like jails, it should be good enough reason not to go with netbsd.
 
cracauer@ said:
With compression on ZFS will be a lot faster for compressible things like the base OS.
Click to expand...
I was thinking along the same lines - perhaps the argument would be to increase compression as to reduce disk footprint along with memory footprint (set highest compression in memory), if say speed isn't the primary factor, but perhaps memory usage plays a larger role in matching UFS without too many changes. Something like that could be ideal, if the cache hit ratio is generally 1 to 1. Anyways - neat stuff :)
 
Had my morning coffee - early this morning while everyone was asleep and had a look a little further and some light reading to scratch the surface. I ended up reducing my VM memory to 512 mb from 4GB and ZFS ran fine racking up memory with some tuned settings. The initial wired memory use seems to be determined by max memory - different than arc memory being reported as used, more than likely pre-fetching memory for future use I would guess. With less memory for the VM wired memory went from 100mb to 85mb.

Tuneable parameters are mostly system wide so lots of tweaks but it would seem people who are tweaking these - from what I've been reading - may have the wrong settings for different pools / filesystems to begin with and some things just are not matching up. Hrad to say.

Noticed even with metadata caching only - the wired memory is very aggressive not arc - either that is some buffering done anyway with ZFS or just arc aggresively compensating for accessing files; Eitherway it seems that memory gets recycled when needed by the main system pretty quick, not too concerned about that.

---

Besides the settings being ideal for general use, I do think it's overkill to start out. I would probably go with no compression on install, base OS probably set primary and secondary cache to use metadata, instead of all. Metadata seems to have a extremely high hit ratio and actual data is next to nothing and not worth caching and or compressing. This is all in regards to OOB install and checking out zfs-stats. Compression can't be rolled back by changing compression settings, I would stick with off by default ...

Just Coffee Talk - enjoy your day everyone! Thanks for all the information.
 
Rolling back on ZFS is normally only done with snapshots/bookmarks by the admin or by ZFS/user when important things get really bad (think disaster recovery without blocks to use in place). The installer could disable compression, copy certain files, then enable it and copy other files to give different compression settings to different files in the same dataset though I suspect a 'zfs sync' type of stage is necessary between the changes. Updates would require the same steps to maintain that configuration and updates while the system is running would then mean other writes may get the otherwise undesired options. Restoring from backup with a single command would not be able to maintain these changes but if off/on changes are surrounded by snapshots then it is possible to maintain that through multiple 'zfs recv' commands that change the dataset properties on each command. Placing different files on different datasets (symlink if needed) is the most reliable+simple way to keep different settings on the files properly. If you are just testing, you can have different datasets you switch between to run comparisons (unclean drives may throw off numbers: magnetic gets slower if data is placed further out and SSD may change speed depending on layout, cache, and free space; consider wiping between tests and using RAM drives). If not using different datasets, you can either script a tool like dd to rewrite a file with its own contents to force it to refresh or use a more specific tool like https://github.com/pjd/filerewrite which was designed for just this purpose. There may be other benefits that get overlooked with such file rewrites and I thought I recalled a feature or 2 (not dedupe/block_cloning) that can cause writing to skip when it is trying to put blocks on disk that already had that data for the file.

If you benchmark it and show improvement (in addition to memory used, try to get time measurement, and if possible an electrical power measurement), then open a PR, you may find something ends up in the installer. One reason to resist it off the top of my head is I think boot environments only manage one dataset without it being branched into many subdivided pieces; please correct me if I am wrong as I have had a hard time learning about layout details that boot environments needed. Old root on zfs wiki pages used to provide a variety of tweaks like stronger compression in log areas, turn it off for distfiles, and also changed other zfs properties besides compression. Doing so required many datasets which meant many parts that cannot be managed by a command that only modifies one dataset at a time. Even so, not everyone uses ZFS for root, nor uses boot environments so other 'improvements' still serve a purpose even if it became a lesser-used option.

View: https://youtu.be/L_kBM_RHa6g?t=276
talks specifically about ZFS caching which goes into detail of how things developed and why they are done as they are. Before that timecode he did mention that compression speeds things up when using an algorithm that is faster than the drive as you successfully shove more data through the I/O pipe that bottlenecks operations but doesn't go into the detail.
 
You must log in or register to reply here.
Back
Top