Let's say i have a PC with X memory and a pool of size Y. How do you dimension the size of the log device (write cache) and cache device (read cache) for this zpool. Guidelines, best practices, tips.
vfs.zfs.txg.timeout
- default is 5 or 10 seconds. Probably times 2 for reserve. That is usually only a couple of GB.Mine is at 85%, and still it does perfectly what I need it for: 1. keep the database mostly in l2arc, 2. hold the metadata for the large-files pool, so that the disks don't start spinning for no purpose only when looked after.Miss ratio of my L2-ARC is 95%
zpool iostat -v
shows throughput on the cache device.I once tried to add one on the desktop, for general acceleration, and found it utterly useless. Investing in a second SSD and run the frequently used filesysteme entirely from them gives a magnitude better results.I added a L2ARC to my storage thinking it might improve performance, but after some time it turned out cache was mostly empty and, like you, a really high miss rate. So it was mostly useless and I removed it again.
Just adding more memory will help a lot too. More room for ARC or filesystem caches (in case of UFS).I once tried to add one on the desktop, for general acceleration, and found it utterly useless. Investing in a second SSD and run the frequently used filesysteme entirely from them gives a magnitude better results.
# zfs-stats -A
------------------------------------------------------------------------
ZFS Subsystem Report Wed Jun 9 13:50:55 2021
------------------------------------------------------------------------
[...]
ARC Size: 42.25% 25.89 GiB
Target Size: (Adaptive) 42.31% 25.93 GiB
Min Size (Hard Limit): 12.50% 7.66 GiB
Max Size (High Water): 8:1 61.29 GiB
Decompressed Data Size: 160.21 GiB
Compression Factor: 6.19
ARC Size Breakdown:
Recently Used Cache Size: 87.93% 22.80 GiB
Frequently Used Cache Size: 12.07% 3.13 GiB
[...]
# zfs-stats -L
------------------------------------------------------------------------
ZFS Subsystem Report Wed Jun 9 13:59:38 2021
------------------------------------------------------------------------
[...]
L2 ARC Size: (Adaptive) 68.31 GiB
Decompressed Data Size: 95.12 GiB
Compression Factor: 1.39
Header Size: 0.94% 915.03 MiB
[...]
L2 ARC Breakdown: 6.03 b
Hit Ratio: 0.12% 7.42 m
Miss Ratio: 99.88% 6.02 b
Feeds: 6.14 m
Think of how a typical desktop is used:
Boot, go into graphical environment, use a browser, use an editor, play some music.
Now keep that in mind with whats been said about the ARC, L2ARC, etc.
Except for what I've said above: It costs memory and with a low amount of frequently read data in your ARC, you won't gain any performance but still reduce your ARC size...L2-ARC is adaptive. So it does not hurt much neither.
CACHE HITS BY CACHE LIST:
Most Recently Used: 11.18% 8.92 m
Most Frequently Used: 88.73% 70.77 m
Most Recently Used Ghost: 0.29% 227.57 k
Most Frequently Used Ghost: 0.58% 460.02 k
That can be estimated as ~1% of the L2ARC size (give or take). That needs to be put into the calculation.One thing that is often missed is: L2ARC also needs RAM for its tables - sometimes A LOT of it. So adding a huge L2ARC effectively decreases the ARC size, thus hurting performance (badly!).
For a database where the working set cannot fit into ram, but can fit into l2arc, it works really good. The tradeoff in ram doesn't matter then, because the ram doesnt fit the usecase anyway, and is best used as l2arc header space.A cache/L2ARC usually makes most sense on big NAS/SANs with a big set of 'hot' data. For most fileservers where only a small percentage of files are accessed regularly, normal ARC in RAM (with maxed out RAM configuration) is more than sufficient.
Yeah, that's pointless. At these figures the data is indeed best accomodated directly in memory, where the ARC will do a much better job in optimizing what is actually needed.[...]
L2 ARC Size: (Adaptive) 68.31 GiB
Decompressed Data Size: 95.12 GiB
Compression Factor: 1.39
Header Size: 0.94% 915.03 MiB
[...]
L2 ARC Breakdown: 6.03 b
Hit Ratio: 0.12% 7.42 m
Miss Ratio: 99.88% 6.02 b
Feeds: 6.14 m
[/code]
NFS is by default 100% sync, it goes all thru the ZIL. (If you want or need that is the other question; I have switched mine to async.)As for separate ZIL devices: this fully depends on the purpose of the pool. The ZIL is only used for synchronous writes, so usually just for some database configurations or VMs. Your normal "fileserver" workload will never touch the ZIL.
ZIL is only for power outages. It is not a cache, it is an "intent log". It doesn't speed writes, it only makes sure that you don't need an fsck when restarting. (And since ZFS does not have fsck, you would be in bad luck if you would need one.)On performance, L2-ARC is adaptive ,so it does not hurt much neither.
On consistancy, if I understood you correctly sko, the ZIL is more fragile to sudden power outages ...
NFS is by default 100% sync, it goes all thru the ZIL. (If you want or need that is the other question; I have switched mine to async.)
noasync
Metadata I/O should be done synchronously, while data I/O
should be done asynchronously. This is the default.
I have 64GB of fast SSD L2ARC cache on my desktop and this seems to be good enough for regular use. The drive is bigger, but I partitioned it that way. Have also swap on the same drive. Depends on the usage profile of course, but in my case it is not even full all the time. Stays somewhere near 60GB. It is good that L2ARC is persistent now with FreeBSD 13.0Let's say i have a PC with X memory and a pool of size Y. How do you dimension the size of the log device (write cache) and cache device (read cache) for this zpool. Guidelines, best practices, tips.
Can answer that later. Not sitting behind my desktop now.Argentum out of curiousity, what is the output of zfs-stats -E under your typical load on that system? Particularly the "CACHE HITS BY CACHE LIST" section, then the MRU/MFU Ghost lines.
L2ARC gets populated by stuff that falls out of ARC, the Ghosts represent "recently fell out of ARC".
Not a problem, it's more of a "Hmm, I wonder what that shows".Can answer that later. Not sitting behind my desktop now.
The only thing right now is that just recently I did some experiments with Chia plotting on my desktop and that almost cleared the entire L2ARC...Not a problem, it's more of a "Hmm, I wonder what that shows".
Yes, for normal mounts. But NFS is different. I did mount, and then I saw all my writes go through the ZIL in full.Isn't the default to only write metadata synchronously and all data asynchronously?
Here it is:Argentum out of curiousity, what is the output of zfs-stats -E under your typical load on that system? Particularly the "CACHE HITS BY CACHE LIST" section, then the MRU/MFU Ghost lines.
L2ARC gets populated by stuff that falls out of ARC, the Ghosts represent "recently fell out of ARC".
------------------------------------------------------------------------
ZFS Subsystem Report Thu Jun 10 17:35:28 2021
------------------------------------------------------------------------
ARC Efficiency: 46.85 m
Cache Hit Ratio: 96.57% 45.24 m
Cache Miss Ratio: 3.43% 1.61 m
Actual Hit Ratio: 95.73% 44.85 m
Data Demand Efficiency: 88.84% 9.20 m
Data Prefetch Efficiency: 36.03% 274.54 k
CACHE HITS BY CACHE LIST:
Anonymously Used: 0.57% 258.46 k
Most Recently Used: 22.74% 10.29 m
Most Frequently Used: 76.39% 34.56 m
Most Recently Used Ghost: 0.28% 124.62 k
Most Frequently Used Ghost: 0.03% 12.59 k
CACHE HITS BY DATA TYPE:
Demand Data: 18.06% 8.17 m
Prefetch Data: 0.22% 98.91 k
Demand Metadata: 80.36% 36.36 m
Prefetch Metadata: 1.36% 616.62 k
CACHE MISSES BY DATA TYPE:
Demand Data: 63.95% 1.03 m
Prefetch Data: 10.94% 175.63 k
Demand Metadata: 15.86% 254.66 k
Prefetch Metadata: 9.25% 148.44 k
------------------------------------------------------------------------
ZFS Subsystem Report Thu Jun 10 16:55:26 2021
ARC Efficiency: 3.88 b
Cache Hit Ratio: 99.97% 3.87 b
Cache Miss Ratio: 0.03% 1.15 m
Actual Hit Ratio: 99.93% 3.87 b
Data Demand Efficiency: 99.27% 50.51 m
Data Prefetch Efficiency: 62.78% 542.10 k
CACHE HITS BY CACHE LIST:
Anonymously Used: 0.02% 905.46 k
Most Recently Used: 0.87% 33.85 m
Most Frequently Used: 99.09% 3.84 b
Most Recently Used Ghost: 0.01% 306.90 k
Most Frequently Used Ghost: 0.01% 253.79 k
CACHE HITS BY DATA TYPE:
Demand Data: 1.29% 50.14 m
Prefetch Data: 0.01% 340.35 k
Demand Metadata: 98.65% 3.82 b
Prefetch Metadata: 0.05% 1.86 m
CACHE MISSES BY DATA TYPE:
Demand Data: 32.07% 367.52 k
Prefetch Data: 17.60% 201.75 k
Demand Metadata: 40.12% 459.88 k
Prefetch Metadata: 10.21% 116.98 k
L2 ARC Summary: (HEALTHY)
Low Memory Aborts: 209
Free on Write: 17.80 k
R/W Clashes: 1
Bad Checksums: 0
IO Errors: 0
L2 ARC Size: (Adaptive) 10.15 GiB
Decompressed Data Size: 12.43 GiB
Compression Factor: 1.22
Header Size: 0.25% 32.40 MiB
L2 ARC Evicts:
Lock Retries: 15
Upon Reading: 0
L2 ARC Breakdown: 1.14 m
Hit Ratio: 52.94% 601.71 k
Miss Ratio: 47.06% 534.91 k
Feeds: 39.38 k
L2 ARC Writes:
Writes Sent: 100.00% 33.08 k
CACHE HITS BY CACHE LIST: Anonymously Used: 0.57% 258.46 k Most Recently Used: 22.74% 10.29 m Most Frequently Used: 76.39% 34.56 m Most Recently Used Ghost: 0.28% 124.62 k Most Frequently Used Ghost: 0.03% 12.59 k
See how small the "...used by ghost" are? That is stuff that would wind up in L2ARC.CACHE HITS BY CACHE LIST: Anonymously Used: 0.02% 905.46 k Most Recently Used: 0.87% 33.85 m Most Frequently Used: 99.09% 3.84 b Most Recently Used Ghost: 0.01% 306.90 k Most Frequently Used Ghost: 0.01% 253.79 k
------------------------------------------------------------------------
ZFS Subsystem Report Thu Jun 10 18:35:47 2021
------------------------------------------------------------------------
L2 ARC Summary: (HEALTHY)
Low Memory Aborts: 6
Free on Write: 4.69 k
R/W Clashes: 0
Bad Checksums: 0
IO Errors: 0
L2 ARC Size: (Adaptive) 55.87 GiB
Decompressed Data Size: 78.85 GiB
Compression Factor: 1.41
Header Size: 0.14% 116.00 MiB
L2 ARC Evicts:
Lock Retries: 4
Upon Reading: 0
L2 ARC Breakdown: 1.61 m
Hit Ratio: 29.55% 475.94 k
Miss Ratio: 70.45% 1.13 m
Feeds: 84.89 k
L2 ARC Writes:
Writes Sent: 100.00% 16.04 k
------------------------------------------------------------------------