ZFS Zfs Tuning

mayers said:
3 3TB disks with which to raid ZFS. The RAM is currently 4GB but I intend to upgrade to at least 8GB.
Click to expand...
mayers said:
So I thought about making 16GB of RAM, 1 SSD operating system, 1 SSD L2ARC cache and 3 ZFS raid disks with active deduplication.
Click to expand...
For deduplication of RaidZ from 3 × 3 TB drives, you need 30 GB of RAM for dedupe ALONE. 48 GB is your starting point.

mer said:
The pfSense stuff default install is ZFS; the appliances sold by Netgate range from 1GB to 16GB. As these are firewall devices, the use pattern of the storage is mostly write, I don't think there is much if any tuning for ZFS.
Click to expand...
I'm almost sure it's not mostly write. I am using m0n0wall - the predecessor of pfSense - and it writes nothing except config file, when you edit it. Logs, stats, ARP tables etc. are kept in RAM. These OSes are designed to be run from SD cards, and SD cards are killed quickly with continuous writes.
 
  • Thanks
Reactions: mer
homeadm said:
I'm almost sure it's not mostly write. I am using m0n0wall - the predecessor of pfSense - and it writes nothing except config file, when you edit it. Logs, stats, ARP tables etc. are kept in RAM. These OSes are designed to be run from SD cards, and SD cards are killed quickly with continuous writes.
Click to expand...
Yes, I worded it poorly. What I was trying to convey is pfSense/m0nowall after boot which is obviously read operations, they don't do a lot of reads that inflate ARC.
 
homeadm said:
For deduplication of RaidZ from 3 × 3 TB drives, you need 30 GB of RAM for dedupe ALONE. 48 GB is your starting point.


I'm almost sure it's not mostly write. I am using m0n0wall - the predecessor of pfSense - and it writes nothing except config file, when you edit it. Logs, stats, ARP tables etc. are kept in RAM. These OSes are designed to be run from SD cards, and SD cards are killed quickly with continuous writes.
Click to expand...
Sorry but the raid of 3 3TB disks should be 2/3 so 6TB I read that 2GB of RAM is needed for every 1TB so 6x2 = 12. Did I read that wrong? Thank you
 
Exact deduplication requirement is 320 bytes for each block:

ZFS Data Deduplication Requirements - Transitioning From Oracle® Solaris 10 to Oracle Solaris 11.2

You can use the deduplication ( dedup ) property to remove redundant data from your ZFS file systems. If a file system has the dedup property enabled, duplicate data...
docs.oracle.com

In typical scenarios, average block size is 64 kB, hence requirement of 5 GB of RAM per every terabyte of storage. Here is a great article about the topic:

ZFS: To Dedupe or Not to Dedupe…

…that is the question. Ever since the introduction of deduplication into ZFS, users have been divided into two camps: One side enthusiastically adopted deduplication as a way to save storage space, while the other remained skeptical, pointing out that dedupe has a cost, and that it may not be always …
constantin.glez.de constantin.glez.de

Ignore calculations of costs, they are outdated.
 
homeadm said:
Exact deduplication requirement is 320 bytes for each block:

ZFS Data Deduplication Requirements - Transitioning From Oracle® Solaris 10 to Oracle Solaris 11.2

You can use the deduplication ( dedup ) property to remove redundant data from your ZFS file systems. If a file system has the dedup property enabled, duplicate data...
docs.oracle.com

In typical scenarios, average block size is 64 kB, hence requirement of 5 GB of RAM per every terabyte of storage. Here is a great article about the topic:

ZFS: To Dedupe or Not to Dedupe…

…that is the question. Ever since the introduction of deduplication into ZFS, users have been divided into two camps: One side enthusiastically adopted deduplication as a way to save storage space, while the other remained skeptical, pointing out that dedupe has a cost, and that it may not be always …
constantin.glez.de constantin.glez.de

Ignore calculations of costs, they are outdated.
Click to expand...

In the article it is in fact indicated to use L2ARC

If performance isn’t critical, and if you expect to save more than 20% of storage capacity through deduplication, then go for it but add at least 5GB of L2ARC SSD capacity per TB of pool data to store the dedup table in.
 
HI !!! :)

I created the zfs pool and inserted a cache disk

NAME STATE READ WRITE CKSUM
datastore01 ONLINE 0 0 0
ada2 ONLINE 0 0 0
cache
ada1 ONLINE 0 0 0

But zfs-stats gives me l2arc not enabled

------------------------------------------------------------------------
ZFS Subsystem Report Fri Jan 5 18:53:05 2024
------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------
 
The problem resolved itself in the sense that the command now returns the correct output, but I didn't change anything

------------------------------------------------------------------------
ZFS Subsystem Report Tue Jan 9 08:58:27 2024
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
Passed Headroom: 0
Tried Lock Failures: 0
IO In Progress: 0
Low Memory Aborts: 0
Free on Write: 0
Writes While Full: 0
R/W Clashes: 0
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 0

L2 ARC Size: (Adaptive) 342.50 KiB
Decompressed Data Size: 10.08 MiB
Compression Factor: 30.14
Header Size: 0.00% 0 Bytes

L2 ARC Breakdown: 102
Hit Ratio: 0.00% 0
Miss Ratio: 100.00% 102
Feeds: 68.58 k

L2 ARC Buffer:
Bytes Scanned: 0 Bytes
Buffer Iterations: 0
List Iterations: 0
NULL List Iterations: 0

L2 ARC Writes:
Writes Sent: 100.00% 6

------------------------------------------------------------------------
 
homeadm said:
they are outdated.
Click to expand...
What about L2ARC? The article you linked says:
So, for deduplicated installations that are not performance-critical, using an SSD as an L2ARC instead of pumping up the RAM can be a good choice. And you can mix both approaches, too.
Click to expand...

How much slower is L2ARC on NVMe compared to in-memory?

I am limited to 64GB on C246 platform. 4x12TB MG07 disks in a mirror stripe
 
Last edited:
In the same regard what about putting L2ARC on the same NVMe as FreeBSD.

I asked the question to blacklion and his suggestion was in regards to FreeBSD OS files that frequently write to disk. They are similar to my findings but I am more concerned about the wear and tear on NVMe with L2ARC.
I can easily backup the OS config files but I was wondering about the dual pounding of mem cache and OS.

Using 2280 NVme Samsung PM9A1 in 512GB.
Code: 
diskinfo -i /dev/nvd0
/dev/nvd0
    512             # sectorsize
    512110190592    # mediasize in bytes (477G)
    1000215216      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    SAMSUNG MZVL2512HDJD-00BLL    # Disk descr.
    S760NE0W634867    # Disk ident.
    nvme0           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Asynchronous random reads:
    sectorsize:    866579 ops in    3.000079 sec =   288852 IOPS
    4 kbytes:      854194 ops in    3.000059 sec =   284726 IOPS
    32 kbytes:     316782 ops in    3.000730 sec =   105568 IOPS
    128 kbytes:     79157 ops in    3.004545 sec =    26346 IOPS
    1024 kbytes:     9962 ops in    3.038214 sec =     3279 IOPS

It runs strong on PCIe 3 bus:
Code: 
Transfer rates:
    outside:       102400 kbytes in   0.042361 sec =  2417318 kbytes/sec
    middle:        102400 kbytes in   0.037407 sec =  2737456 kbytes/sec
    inside:        102400 kbytes in   0.037199 sec =  2752762 kbytes/sec

Will it last? I would like to have L2ARC on Enterprise drive.
 
I don't plan on using dedup so 64GB should be OK with just ARC.
I will keep my eye on it.
Thanks for the pertinent article. I wish it had a date.
 
I can add dedup to the pool later right?

Maybe get a bigger enterprise NVMe in the future for L2ARC if it is needed with 64GB max on this Motherboard..

That might be a good time to consider Boot Environment and ditch NVMe with UFS for OS and use it strictly for L2ARC.
 
A bit surprising that a source of pretty useful info didn't have timestamps... I expected better.

Some places on klarasystems.com do have timestamps, but among those that should - most don't.

This is probably one of those things that do get ignored unless we all get obnoxiously blunt and loud about it. 😩

Call me old-school, anal, obtuse - but timetamps are important, dammit...
 
astyle said:
… Some places on klarasystems.com do have timestamps, but among those that should - most don't.

This is probably one of those things that do get ignored unless we all get obnoxiously blunt and loud about it. 😩

Call me old-school, anal, obtuse - but timestamps are important, dammit...
Click to expand...

astyle I'm with you on the importance.

When I last raised the issue, in Discord, I injected some humour, because me asking more than twice could be perceived as ungrateful nagging.

kevans I hope you don't mind me quoting here (emphatically: no onus on you or anyone to make or drive the change):

1705460501439.png
 
You must log in or register to reply here.
Back
Top