ZFS zfs memory requirements

decuser

decuser

I'm setting up a fossil server on a raspberry pi 3 wtih 1GB of RAM. I have read that ZFS is memory hungry and that a minimum of 1GB is recommended. I have also read that ECC is required. My question is this, does ZFS really need ECC, and does it really need more than 1GB in order to operate without breaking down (panics, corruption, etc), or is it just a matter of worrying about performance.

My motivation for asking is I've used ZFS non-stop for ever and never had a non-recoverable error (sure, I've had drives fail, but nothing that caused a real problem). But I've always run it on a machine with 8GB of RAM. I'm gonna try it, but the thought of memory related issues that could happen without any warning is a little bit scary and I though I'd ask the forum denizens for hopefully more informed answers.
 
With 1GB of ram i should stick with ufs.
No, ECC is not a requirement.
My PC, running ZFS, on which i write this message has no ECC memory.

Note , you can limit zfs-memory usage with :
/etc/sysctl -> vfs.zfs.arc_max = ...
 
Easy question first: Is ECC required? No. But it is a good idea.

Here's why. ZFS is memory hungry, more than other file systems. Given the way it writes data to disk (non-linear, log-structured), it tends to have a large amount of metadata in memory, and make heavy use of that metadata. If that metadata gets corrupted in memory, bad things will happen. An even more important argument is the following: ZFS always reads and writes CRCs, so the data on disk and on the way to/from the disk is safe against corruption. That means that data in memory is now the largest target for corruption. While this doesn't mean that ECC is more necessary than for other filesystems (which also fill all unused memory with cache data), for ZFS the relative importance of memory errors is higher.

Having said that: I run at home without ECC; it simply wasn't available in the desired form factor when I bought my server.

On the amount of memory: 1 gig is SUPER CRAZY INSANELY tight. I have 3 GB at home (the hardware has 4, but because of limitations of the x386 only 3 can be used), and I have to set a few sysctls to make it work, and it isn't' fast. If you look around on this forum, you'll find what you have to set. But please don't set your expectations too high; it's quite possible that ZFS with 1 gig will be a performance dog, and UFS (or ext4) will just work more smoothly.

One more thing to remember: ZFS is a bit of a CPU hog. The single biggest reason is that it always has to calculate CRCs (duh). On modern CPUs, that's de-facto free, because there is so much CPU power compared to the IO speed of disk drives. Matter-of-fact, on a modern system investing CPU power for better data reliability is a great choice, and the ZFS design is sensible. On a RPi with just 4 cores each running at below 1 GHz, you might get frustrated waiting for the data.
 
Wow, was that a quick test! Hangs and slow and and and.. Just say NO :). I hooked it all up, did some loader.conf magic and ew is the best I can say. Thanks ralphbsz and Alain, great tips. I'm sure it'd work fine on my rpi4 with 8gigs, but the rpi3 with 1gig, no.
 
I am another one of those who uses ECC whereever he can. But saying that it is required for ZFS is a misnormer. If anything the internal checks of ZFS make it slightly safer to run free errors for all (non-ECC).

The amount of memory required is higher than FFS, especially if your applications write mmap'ed data.
 
Last edited:
I think that the answer lies on how you intend to use the machine. I'm typing this on a FreeBSD VM with 2 GB memory configured.
It's on my notebook, and there's no option for ECC. The VM is only used for email and web browsing when I'm away from the office.
I'm running minimal system load, plus sendmail plus one small chrome. You can see "772M Free". So, I'd be into swapping with only 1 GB:
Code: 
# top -d1 0
last pid:  4162;  load averages:  0.87,  0.55,  0.41; b up 0+20:05:33  14:30:18
36 processes:  1 running, 35 sleeping
CPU:  0.4% user,  0.0% nice,  0.7% system,  0.1% interrupt, 98.8% idle
Mem: 211M Active, 433M Inact, 29M Laundry, 507M Wired, 772M Free
ARC: 164M Total, 60M MFU, 91M MRU, 95K Anon, 1300K Header, 11M Other
     120M Compressed, 175M Uncompressed, 1.45:1 Ratio
Swap: 2048M Total, 2048M Free
I think you would struggle with response on a Pi if it starts to swap...
 
As gpw928 has already said, it matters a lot how you are going to use it. I couldn't immediately find if you want to use it as a server or as a desktop with GUI. As a server with a light workload, 1GB RAM will still suffice. But if you want to run a GUI, you might want to choose UFS.

I have often read that UFS is a much less robust file system than ZFS in many areas. I also want to say that info and recommendation are not always correct. Eg TrueNAS say 16GB RAM is required. But I can easily run it in VirtualBox on FreeBSD with only 2.4GB of RAM allocated. And then it runs TrueNAS and Nextcloud both and it runs very stable. Their advice is probably for professional customers who want to run high workloads and want a lot of performance.

Another error can be found e.g. in the ZFS documentation: https://github.com/openzfs/zfs/releases
  • Linux: compatible with 3.10 - 6.0 kernels
This documentation is wrong as I am currently running this version on Clear Linux with kernel 6.1.0-1221.native
 
decuser said:
I'm setting up a fossil server on a raspberry pi 3 wtih 1GB of RAM. I have read that ZFS is memory hungry and that a minimum of 1GB is recommended. I have also read that ECC is required. My question is this, does ZFS really need ECC, and does it really need more than 1GB in order to operate without breaking down (panics, corruption, etc), or is it just a matter of worrying about performance.

My motivation for asking is I've used ZFS non-stop for ever and never had a non-recoverable error (sure, I've had drives fail, but nothing that caused a real problem). But I've always run it on a machine with 8GB of RAM. I'm gonna try it, but the thought of memory related issues that could happen without any warning is a little bit scary and I though I'd ask the forum denizens for hopefully more informed answers.
Click to expand...

Check the most common myths about ZFS:

zgXfcCC.png

... also when it comes to ECC - ALL filesystems benefit from having ECC memory - but ZFS having all its protections and checksums is the filesystem that is BEST without ECC. In other words - you do not need ECC for ZFS (or any other filesystem - but ZFS copes best on systems without ECC from all available filesystems. Its just another myth that ZFS 'requires' ECC RAM.

Same for memory usage. You can limit ZFS memory usage to your needs. If you limit it to almost zero - then ZFS behaves like every other filesystem - just using the 'performance' of the underlying devices - but if you have 'some' RAM then ZFS will use it to cache as much as possible to 'do not touch' the disks.

Hope that helps.
 
The problem with Pi3/1GB RAM is not only the size of RAM but slow CPU/small cache. I hope you are using 7200rpm HDD and not SD.
 
What 6502 and gpw928 already said! If using RAM for ZFS means that less RAM is available for other processes, and those start swapping, and they swap onto ultra-slow SD card, that is going to be painful.
 
ralphbsz said:
What 6502 and gpw928 already said! If using RAM for ZFS means that less RAM is available for other processes, and those start swapping, and they swap onto ultra-slow SD card, that is going to be painful.
Click to expand...

It will also wear out the SD card.
 
You must log in or register to reply here.
Back
Top