Networked ZFS replication target: Is it crazy to use a RPi model B?

floogs

floogs

To briefly summarize my end goal: I have a FreeNAS 11.2 machine with a RAID-Z1 (4x6TB) storage pool that I'd like to backup to a physically external volume (i.e. somewhere else in the house). I picked up a couple of cheap 10TB USB3 drives w/ external power sources that I plan to stripe, since redundancy isn't a big concern for a secondary backup. I have a Raspberry Pi model B (512MB RAM version) that I was thinking about trying to use to run the ZFS pool that the FreeNAS snapshots will replicate to, but first a couple of questions:

1. Is this sane to even attempt with the Pi model B's very modest horsepower? I did get a kernel warning about recommended memory for ZFS as follows:

Code: 
ZFS WARNING: Recommended minimum RAM size is 512MB; expect unstable behavior.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
                         Consider tuning vm.kmem_size and vm.kmem_size_max
                         in /boot/loader.conf.

Between that and the recommendations in the wiki article about ZFS tuning, I'd guess it's foolish to even try this on 512MB (actually less, since the Pi allocates some system memory to the GPU and I haven't found how to modify that amount in FreeBSD). But I wondered if the workload in my scenario might be light enough to get away with it.

(I'm not averse to picking up a Pi 4 for this purpose which almost certainly has enough oomph to do the job, but it's fun to put old hardware back into service when possible. I saw the recommendation to use the Pi 3's arm64 image for the 4 -- how is compatibility so far for anyone who's tried it?)

2. Once I settle on which Pi to use: Are ZFS pools created in FreeNAS compatible with FreeBSD and vice versa? I ask because copying several terabytes should go a lot faster over USB3 than over the network (even with the Pi 4's gigabit ethernet), so I was hoping to hook the drives straight to the FreeNAS machine, create the pool there, seed the initial backup, then move them to the Pi and import the pool into FreeBSD and run nightly incremental backups from there. Would that work?

Sorry if any of this is overly obvious or rudimentary, my experience with ZFS and FreeBSD is limited to about a year with FreeNAS.
 
There is lots of information on the web about the minimum amount of memory required for ZFS. A lot of that is wrong. For example, they say that you need so-and-so many GB of RAM for each TB of disk space, that you need a 64-bit CPU, and that you need ECC RAM. All of these statements contain a grain of truth, but taken as absolutes they are just wrong. As an example, I run a ZFS server with about 4 or 5TB of managed disk space on a 32-bit machine with 3GB of RAM and no ECC, and it mostly works. However, running 20TB of ZFS space with 1/2GB of RAM is extraordinarily far away from optimal, and it might just not work. By all means, try it, but don't get surprised if you need to buy an RPi4 with 4GB.

Because of the checksums (which are one of the best and most important features), ZFS tends to consume quite a bit of CPU power. I don't know what your throughput is going to be on a relatively weak CPU like an RPi. If it is too little, two things will happen: It will get slow, and your CPU will get hot. So I would start by attaching a heatsink to the CPU chip, and then measuring. On the other hand, on the old RPIs, you only have 10MByte/s of network throughput (slow ethernet), so maybe speed is not a problem.

About compatibility of ZFS between nodes: Yes, but. You need to look which version of FreeBSD is really running underneath FreeNAS, and then look at the versions of ZFS and the feature flags, and format your file system to use only the common subset of features between the two OSes. This will probably add an hour of reading documentation and configuring feature flags, not a big deal.
 
Thanks for the info, this is really helpful.

You're right that the Pi only has 10Mbit ethernet, and the hard drives would be using USB2 ports, so throughput would already be extraordinarily bottlenecked which might allow the CPU to breathe a bit. That said, a dramatically more powerful Pi 4 is a no-brainer for a measly ~60 bucks -- though it looks like Pi 4-specific hardware support is all but nonexistent as of right now. Do you have any sense of how long support for new hardware usually takes to come online? From glancing around this forum I gather the Pis aren't as popular in the community as some of the more powerful hobbyist SBCs so I assume I may be waiting a bit.

As for configuring compatibility with FreeNAS (I'm on 11.2, which is itself built on FreeBSD 11.2-STABLE), this looks pretty detailed -- is this a comprehensive resource or can you think of anything else I'd need to read up on before proceeding?
 
In that case, the easiest would probably be to install FreeBSD 11.2 on the Pi, pretty much guaranteed to have the same feature flags.

The question of FreeBSD support for RPi's is always hotly debated here. Right now, it is volunteer-only tier 2 support, and things kind of mostly work, with some sad exceptions. You might get lucky, and you might waste your time. I ran FreeBSD on a Pi for a few weeks, until the various limitations and problems made it more efficient for me to switch to Raspbian.

So here is an alternate proposal: The Linux version of ZFS is disk-format compatible with the FreeBSD = FreeNAS version. And in the future, the two versions will grow even closer together. Looking around the web for 30 seconds, there are mentions of running ZFS on Raspbian. Maybe you should try that?
 
One other thing to keep in mind: on all Raspberry Pi's before Pi4 the network controller is connected to the same single usb controller as the usb ports. This is not optimal for a network - to disk device (which is what a backup server is). You would be wise to try it and see if you are happy with the performance before you commit to the solution.
 
I tried it with Linux and a Pi2 with 4x4TB and immediatly cancelled this little private project ... it did not work
 
What does "it did not work" mean? Couldn't get it configured? Hardware didn't function? Throghput too slow? It kept crashing? Power consumption too high?
 
Was this with ZFS (a.k.a. ZoL)? That would be strong evidence that 512MB of memory is *not* enough for ZFS with this many disks. Which we were suspecting anyway.
 
yep, it was raspbian with ZOL ... some devs told me if it can import the pool low memory should be fine. however it is not. but this was about 18 months ago. hope freebsd will soon be possible on a pi4 ;-)
 
floogs said:
Between that and the recommendations in the wiki article about ZFS tuning, I'd guess it's foolish to even try this on 512MB (actually less, since the Pi allocates some system memory to the GPU and I haven't found how to modify that amount in FreeBSD). But I wondered if the workload in my scenario might be light enough to get away with it.
Click to expand...

It's not a workload problem. It's a problem of precisely tuning the memory usage.
The workload is what ZFS decides it wants to do just now, and probably nobody does understand that fully, other than that it usually works, except when it hits some memory limit. You cannot really influence this, you can only adjust the memory via trial+error so that it does not hit limits.

I tried it, early when ZFS appeared in FreeBSD. I had a database (~10GB) on a machine with 256MB RAM, and I wanted that to run from ZFS due to copy-on-write - performance didn't matter. 256MB does not even start. 384 MB did work, but it was a pain, kept hitting limits and could not be properly used. 768MB did work, with careful tuning.

But that was 10 years ago, and data structures probably have grown. So, 512MB minus X might be possible to get working, but it will be difficult, probably unstable, and not fun.

Later I expanded the ZFS over the whole system. It did not appear that ressource consumption would be much dependent on the number of pools, filesystems, or disks. But tens of TB might still be a different matter.

Here are some of the options I was playing with - no idea anymore about which of them did do any good, and probably they have changed in the meantime:
#vfs.zfs.vdev.cache.bshift="14"
#vfs.zfs.zfetch.array_rd_sz="131072"
#vfs.zfs.zfetch.block_cap="32"
#vfs.zfs.zfetch.max_streams="3"
#vfs.zfs.vdev.max_pending="10"
#vfs.zfs.arc_meta_limit="14M"
 
To resurrect this thread a bit, I just got my hands on a spare Raspberry Pi 2, which is a (meager) quad core with 1GB of RAM, so I'm looking at giving this a shot now that I have a little more compute power and memory cushion on hand. (The Pi 4 4GB model has been out of stock everywhere since release, so that option is on hold for now.)

I'm not averse to trying Raspbian with ZFS on Linux instead of FreeBSD -- I assume all the same options for tuning memory usage are available there? I've been looking at this guide which claims to have successfully run ZFS on a 512MB Pi. Regardless of which OS I settle on, would it be sensible to simply double his values as a starting point, or is that an overly simplistic approach?

On a side note, the 12.0-STABLE image I'm running on the Pi 2 disables the power and activity LEDs on the PCB (or more specifically seems to subsume them into some sort of GPIO interface). It's only a minor annoyance, but I couldn't find a way to re-enable their default behavior. Does anyone know a way to do that? I tried to search the forum here but "LED" isn't an easy term to filter for, and I'm not sure if it's worth starting a new thread for.
 
You must log in or register to reply here.
Back
Top