Should I / How do I mirror ZFS between two systems

[decuser]

So, I have a couple of identical (or quite nearly so) systems - Lenovo ThinkCentre m92p's, that I would like to put to use. I'd like some opinions on this idea...

I have a directory 0wa that contains all of my slow changing directories and files. This is separate from my home dir ~. It weighs in at about 700GB of data. I usually just share it around via sending a zfs snapshot that's taken whenever I want to bring up a new system. What I wind up with is 8 different versions of the directory that I then have to sync or decide which to keep as master. I'm thinking, why not put it all on my main freebsd machine on a zfs pool that I then share over nfs. I've never tried this, so I'm not sure if it'll work well with multiple systems mounting it over nfs, so if you know something about this, please do tell. But, assuming that that'll work, I'm wondering if I can mirror the zfs pool between two systems or get some kind of minimal lag replication. Here's the gist of it:

odin
-----
freebsd 14 (4 core 3Ghz i7,16GB RAM, upgradable to 32... guess where the other 16 went)
host the zfs pool (mirrored over a couple 1 TB SSDs)
share the pool over nfs

baldur
----
freebsd 14 (4 core, 3Ghz i7, 16GB RAM, upgradable to 32)
mirror (or replicate) the odin pool

tux
----
linux mint 22 (8 core, 16 thread, 96GB RAM) laptop
mount the nfs share and use the dir and files as needed

astra
-----
linux mint 22 (4 core , 2.6 Ghz i5, 16GB RAM) laptop
mount the nfs share and use the dir and files as needed

hmm... as I write, I think, does nfs work for laptops... is it graceful when out of range / offline.

In summ:

1. Is is practical to mirror a pool over two systems?
2. Can I mount the pool and use it on multiple systems using nfs?
3. Will the laptops handle it gracefully?

Thanks!

Will

 

[mer]

1. Is is practical to mirror a pool over two systems?
2. Can I mount the pool and use it on multiple systems using nfs?
3. Will the laptops handle it gracefully?

My opinions:
Not sure what you mean about "mirror a pool over two systems"
Mounting the pool and using NFS to to use on multiple systems: absolutely doable, my opinion likely the best solution
Laptops: should handle NFS remote mounts just fine. Just keep in mind that it all depends on the NFS server being available. If the server is on your home network and you take the laptop to work, the data won't be available.

hmm... as I write, I think, does nfs work for laptops... is it graceful when out of range / offline.

This part:
Take a look at "automount". Basically automount configuration says "when this client (laptop) cd's to the following directory, make sure the NFS share from (server) is mounted"
If you move the laptop to $WORK, and you try to access that directory, you are going to get errors or maybe some temporary hangs. As long as you keep that in mind, NFS and automount works just fine. In fact that was the standard configuration in a lot of companies back in the 1980s/1990s. NFS share of home directories, shared machines in a lab that automounted $HOME based on your user login.

 

[decuser]

My opinions:
Not sure what you mean about "mirror a pool over two systems"
Mounting the pool and using NFS to to use on multiple systems: absolutely doable, my opinion likely the best solution
Laptops: should handle NFS remote mounts just fine. Just keep in mind that it all depends on the NFS server being available. If the server is on your home network and you take the laptop to work, the data won't be available.

Very helpful. Thanks.

This part:
Take a look at "automount". Basically automount configuration says "when this client (laptop) cd's to the following directory, make sure the NFS share from (server) is mounted"
If you move the laptop to $WORK, and you try to access that directory, you are going to get errors or maybe some temporary hangs. As long as you keep that in mind, NFS and automount works just fine. In fact that was the standard configuration in a lot of companies back in the 1980s/1990s. NFS share of home directories, shared machines in a lab that automounted $HOME based on your user login.

This sounds great. I'm not wanting to mount $HOME over NFS, but the idea is the same. Off to read up on automount.

 

[Erichans]

What I wind up with is 8 different versions of the directory that I then have to sync or decide which to keep as master

Laptops: should handle NFS remote mounts just fine. Just keep in mind that it all depends on the NFS server being available. If the server is on your home network and you take the laptop to work, the data won't be available.

Don't know enough about your usage patterns, but it sounds like you often come across generating data "off-line", i.e. away from your designated main (ZFS) server, like the laptop example and find yourself into a situation that you'd have to "resync". I don't have any practical experience with such a situation where it concerns non-source code repositories, but it does have a certain likeness to a distrubuted versiontrol control system where people (~laptops) generate and adapt data (off-line) on their own and have to sync that (actually merge) with the central repository. Maybe others have experience or dabbled with such kind of a solution.

 

[mer]

Erichans different thought process than mine. Source code systems like git are very much the model you are thinking of.
If it were me, the server would be running say the git server, the laptops having git repos so the user can do work locally and then simply push changes to the git server. Basically what I do for $WORK. Maybe a bit more about the OP usage patterns we could help.
Things like git are good because everyone thinks about "source code" but they can be used for almost anything.

 

[bgavin]

I use a different method, but with single disks.

I do a one-way update of changed/new data from Source Disk to Destination Disk.
The reason is any accidental delete on Source is NOT deleted from Destination.

Somewhere along the line, I lost most of my photo work from 2019.
The Source disks were a mirrored pair, so the accidental delete was applied to both disks.
I didn't catch it until 2024... no idea it even happened.

Due to the huge size of my photo work, a backup scheme wasn't in the cards for me.
I did do periodic backups to various external USB disks and did recover the vast amount of missing work.
Pure luck on my part.

This got me to thinking about the asynchronous one-way copying of Source to Destination.
This results in Destination clutter when files or directories on Source are renamed then copied to Destination.

The downside to any online scheme is data corruption by malware or ?? is duplicated from Source to Destination.
To keep backup space within reasonable limits, I do a Differential only backup these days.
The old 21 dataset backups I learned in my Novel days is simply too large.

 

[ralphbsz]

If the two machines are physically close, both very reliable, and on the same power feed (so one doesn't go down when the other one is up), you could put one disk drive in the primary machine, and a second drive in the backup machine. Then use a block protocol like iSCSI for the backup machine to make that disk available to the primary machine. On the primary machine, you then configure the pool to be a mirror between two boring zvol volumes, one of which happens to be connected via network. With Gigabit ethernet, the performance will be acceptable; with faster stuff (2.5G or 10G are commonly available today), the network completely stops being the bottleneck.

You want the file system to be available on the backup node too? Easiest way to do that is to run an NFS server on the primary, and share it to the backup one this way. This gets a bit crazy if you have intense write traffic from the backup node, since the data will take multiple trips back and forth.

In reality, I would not recommend this. Because there are a few hairs in the soup. First, of either of the nodes fails or goes down, things go bad. Not data loss bad, but you might lose access, and ZFS might afterwards waste lots of time on needless resilvering. Even a simple "shut backup node down for OS upgrade" might cause such things. Second, performance might get pretty ugly if the workload reads and writes from the wrong side, and it will in general be not so great. Another aspect of that is: You need to teach ZFS on the primary node to prefer reading the local disk, not the remote disk, and I don't know how to do that (for writing, it needs to write to both).

In theory, what you really want is called a cluster file system, where both primary and backup node can read and write the disks and expose a single, unified file system. Such things exist, but I don't know of any free version that is reasonable to administer in a home setting. If you go that route, you also have another problem: With just two nodes, a shutdown or failure of either node will bring a cluster file system down, since it can not distinguish a normal shutdown (power for one was unplugged) from a split brain situation due to network partition. In most practical cases, you would need to either enlarge the cluster to 3 nodes, or add have quorum nodes, witness disks, or other complex workarounds, and for home use, that's just too insane.