ZFS Questions about ZFS on FreeBSD and Linux

bugzeo

Member

Reaction score: 18
Messages: 95

First, my idea is to create an encrypted ZFS pool on USB hard drive (using SATA to USB 3.0 converter). My questions are:
  1. Is there a single ZFS implementation of the filesystem for FreeBSD? There's the Sun/Oracle version and more recently the OpenZFS "version". Are those the same in terms of perfect interoperability, or one pool created with one can have issues mounted with the another driver? Can I choose which version so use on FreeBSD or there's only one? If there's only one version which one it is?
  2. A filesystem created by OpenZFS on FreeBSD can work perfectly fine on Linux and vice-versa?
  3. What are the recommended setting for creating a pool/dataset to backup files from Windows? Encryption, encoding (if any) for filenames, maximum filename length and subdirectories depth, auto-generate or not hashed for files, etc.
  4. If the SATA spinning drive is connected to USB using external case, what about S.M.A.R.T. and disk integrity capabilities? Is is required or mandatory to connect the drive to internal SATA port? That wouldn't be possible for 3.5" drives on certain notebooks, max 2.5".
 

Alain De Vos

Daemon

Reaction score: 613
Messages: 2,073

1., 2. Using an encrypted filesystem with Linux & FreeBSD and mixing is living a dangerous life. My 5-cents.
3. Zfs specification.
4. If smart is detected by FreeBSD any connection is good.
 

gpw928

Aspiring Daemon

Reaction score: 220
Messages: 527

If the SATA spinning drive is connected to USB using external case, what about S.M.A.R.T. and disk integrity capabilities? Is is required or mandatory to connect the drive to internal SATA port? That wouldn't be possible for 3.5" drives on certain notebooks, max 2.5".
I use USB to SATA adapters with a pool of external naked 3.5" disks to back up my ZFS server. The disks are stored in anti-static packets in padded boxes. When deployed, they just sit on their anti-static packet on top of the server. They are identical makes and models to the disks inside the server -- and there's enough so that I can quickly replace any internal disk that fails.

SMART works on all of the drives, regardless of the connection method, though for the USB drives smartctl(8) needs the "--device" option to identify the USB to SATA bridge in use.

You need to match the USB standard of your adapter and your USB 3 ports for compatibility. My ZFS server has USB 3.2 Gen 2 ports. My StarTech adapters use USB 3.1 Gen 2 (which are compatible). They work with both 2.5" and 3.5" SATA drives.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

You need to match the USB standard of your adapter and your USB 3 ports for compatibility. My ZFS server has USB 3.2 Gen 2 ports. My StarTech adapters use USB 3.1 Gen 2 (which are compatible). They work with both 2.5" and 3.5" SATA drives.
If my computer has USB 3.0 it would be compatible wouldn't? Also, if the hard drive requires 10w of power, not all ports can supply that, so that StarTech adapter does have additional power port?
 

mtu

Active Member

Reaction score: 111
Messages: 166

  1. All FreeBSD releases until 12.2 use an "OpenSolaris/Illumos heritage" version of ZFS, which is incompatible with "ZFS on Linux" for certain features and pool versions. However, both of these families of code have been unified now: FreeBSD 13.0 uses "OpenZFS 2.0", which is the same code as "OpenZFS 2.0" on Linux, and so the two should be perfectly compatible (in theory, I haven't tested this).
  2. Yes (in theory, I haven't tested this), in the following cases: Both systems are using the same code (see 1.), or the pool version is "old enough", and no incompatible features are enabled (see zpool-features(5)).
  3. My opinion: No matter how much care you take, some Windows-specific metadata ("file attributes" or whatever they're called) could still be lost. You might consider packing the data on the Windows side (zip or something), then just storing the archives on ZFS.
  4. It can work, but it's a lottery. Every USB port, SATA adapter and HDD (and any combination of the three) can behave differently. You just have to test and see.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

So what is the reason FreeBSD switched from OpenSolaris to OpenZFS 2.0? Is that is not being actively developed anymore, or is because of licensing or other issues?
Also, what happens if you upgrade from FreeBSD 11 to FreeBSD 13, and you had drives in ZFS already? Do they break compatibility or are they converted, or do they keep both versions?
 

6502

Active Member

Reaction score: 35
Messages: 112

Why you need ecrypted ZFS on external drive? Why not encrypted UFS or even FAT32 (or other file system) - it seems this will be backup.
 

mer

Well-Known Member

Reaction score: 290
Messages: 488

"A reason" for the switch, based on my understanding of what I've read in the mailing lists, is OpenZFS was always upstream of the FreeBSD implementation. Changes were pushed upstream to ease future work.
"ZoL" (ZFS on Linux) was also downstream of OpenZFS, but I believe that it was more active in fixing bugs and adding features. Native encryption is one such thing.
OpenZFS effectively rebased to ZoL, so FreeBSD had a choice: maintain what they had and diverge from upstream, making bug fixes and updating harder or "rebase" to upstream.
Lots of discussion over on freebsd-arch and freebsd-hackers, the decision was to rebase to OpenZFS for FreeBSD-13.0
I don't know what OpenSolaris/IllumOS decided to do, but I believe that all OSes using ZFS were rebasing to OpenZFS.

If you upgrade from 11 to 13, FreeBSD-13 should be able to understand the zpools created with 11. If you aren't using any features or properties in 11 that don't exist in 13, you should be fine. Just don't run zpool upgrade on the pool until you are sure that you won't try to use them on 11 anymore.

You may need to update the boot loader on your boot devices with the one from 13 (gptzfsboot if you are using ZFS on root and Boot Environments), but I think that should be able to boot the zpools created in FreeBSD-11. Backwards compatibility may be a pain at times, but is easier, you just need to test it.

6502 "Maybe because they want to?" :) Sounds like they may to share it between multiple systems, maybe in an insecure environment that they don't want to worry if they lose the drive.
 

mtu

Active Member

Reaction score: 111
Messages: 166

So what is the reason FreeBSD switched from OpenSolaris to OpenZFS 2.0? Is that is not being actively developed anymore, or is because of licensing or other issues?
It's a long story. The very short version is that in the past few years, "ZFS on Linux" advanced faster and became better maintained than the "OpenSolaris/Illumos heritage ZFS" that FreeBSD was using.

OpenZFS 2.0 is the result of merging the two into a single codebase that unites Linux and FreeBSD. There is universal consensus that ZFS on FreeBSD 13.0 is now better than before.

If you want to know more, you can find all sorts of comprehensive talks on YouTube or podcasts about ZFS history and the OpenZFS effort, and you may want to read this:

Also, what happens if you upgrade from FreeBSD 11 to FreeBSD 13, and you had drives in ZFS already? Do they break compatibility or are they converted, or do they keep both versions?
Your ZFS will continue to work with both FreeBSD 11 and FreeBSD 13, but it will not use all the new features (like encryption) until you zpool-upgrade(8). After you've done that, it will be possible to use the new features, but no longer work with FreeBSD 11, and you may run into boot errors unless you've correctly upgraded your bootloader (just search the forums, it's happening to everybody).
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

Why you need ecrypted ZFS on external drive? Why not encrypted UFS or even FAT32 (or other file system) - it seems this will be backup.
Well, isn't ZFS better than UFS, not only for file integrity, but also detecting hard drive errors?
 

6502

Active Member

Reaction score: 35
Messages: 112

6502 "Maybe because they want to?" :) Sounds like they may to share it between multiple systems, maybe in an insecure environment that they don't want to worry if they lose the drive.
I don't ask why encrypt but why ZFS on external drive. I will prefer to share FAT32 volume between different OS than more complex file system. FAT32 is simple and there is very low risk from incompatibility or similar issues.
 

gpw928

Aspiring Daemon

Reaction score: 220
Messages: 527

If my computer has USB 3.0 it would be compatible wouldn't? Also, if the hard drive requires 10w of power, not all ports can supply that, so that StarTech adapter does have additional power port?, but I'm not an expert.
I am not an expert, but I believe that USB 3.2 devices are generally backwards compatible with all existing USB products. But using mismatched standards may lose (a lot of) speed...

The StarTech adapters I use have an external power supply.
 

mer

Well-Known Member

Reaction score: 290
Messages: 488

I don't ask why encrypt but why ZFS on external drive. I will prefer to share FAT32 volume between different OS than more complex file system. FAT32 is simple and there is very low risk from incompatibility or similar issues.
I understood your question, I stand by my answer :)
 

mer

Well-Known Member

Reaction score: 290
Messages: 488

Well, isn't ZFS better than UFS, not only for file integrity, but also detecting hard drive errors?
ZFS has checksums all over the place on just about every block. So if when reading something from the device, the calculated checksum does not match the stored checksum, then yes, it may be an indicator of drive errors. You should then run smartmontools on the drive to see what it says.
But RAM or CPU errors could also cause a difference in the calculated vs stored checksums.

UFS does a pretty good job with file integrity; there are occasional bugs or combinations of flags that affect things, but that is pretty common with all filesystems ever created. It's how you recover from errors that is important.
 

grahamperrin

Daemon

Reaction score: 535
Messages: 1,731

… FAT32 is simple and there is very low risk from incompatibility or similar issues.

Not great for integrity of data, and (if I understand correctly) two copies of the FAT is not a panacea when problems occur.

… how you recover from errors …

Edge/extreme case: <https://lists.freebsd.org/archives/freebsd-current/att-0339/2021-07-16_00.53_typescript.txt>

… what is the reason FreeBSD switched from OpenSolaris to OpenZFS …

FreeBSD and ZFS | FreeBSD Foundation (2016-03-04)

<https://issue.freebsdfoundation.org/publication/?i=706299&ver=html5&p=25>

et cetera
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

Why you need ecrypted ZFS on external drive? Why not encrypted UFS or even FAT32 (or other file system) - it seems this will be backup.
This is because I may be backing up data from drives encrypted with BitLocker from Windows, I can't compromise data from third parties. Another option would be to use dislocker on the external drive, but I don't think there's a port available for FreeBSD/NetBSD/OpenBSD.

Are FreeBSD-UFS encrypted volumes able to be read/written by Linux?
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

I don't ask why encrypt but why ZFS on external drive. I will prefer to share FAT32 volume between different OS than more complex file system. FAT32 is simple and there is very low risk from incompatibility or similar issues.
FAT32 doesn't support files bigger than 4GB, error handling in case of unexpected shutdown is really poor, it can cause problems with long paths and filenames, and doesn't defragment automatically like NFTS does. FAT32 is for usb sticks and generic SD cards where you store simple stuff. Even smartphones nowadays exFAT instead.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

I am not an expert, but I believe that USB 3.2 devices are generally backwards compatible with all existing USB products. But using mismatched standards may lose (a lot of) speed...

The StarTech adapters I use have an external power supply.
I see it has a 12V barrel connector. I was expecting it to have already USB-C or something more modern, so I can just plug the any USB-C cable connected to a wall charger. At least says that the AC adapter is included on Technical Specifications, which is awesome.

What do you think about HDD Docking Stations from same brand? The USB 3.1 dual-bay is quite expensive.
 

Alain De Vos

Daemon

Reaction score: 613
Messages: 2,073

When you do encrypting stick to one operating system. My 5-cent.
And if you do encrypting make sure the underlying filesystem is simple , maybe fat32 or ufs.
Some linux distributions are capable of reading ufs.
Freebsd can read-write xfs but that is own risk.
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

Just chiming in here... I'd stick to ZFS over UFS for base OS installation. Reason for that - I can adjust directory size limits (like limiting /usr/home to no more than 20GB, but making sure it's got no less than 10 GB) later. I also like the snapshotting feature of ZFS (although rolling back a ZFS snapshot is not completely intuitive).

FWIW, you can add another device (Even over USB or NFS) to a ZFS pool. I'd recommend treating that device as a completely separate dataset, though. On FreeBSD it's enough of a headache to mount a USB device properly. I'd prefer to skip the complexity of using ZFS for the external device. /bin/cp is not gonna care about that anyway.

But generally, I'd suggest avoiding doing fancy and complex stuff just because it's possible. Lining things up to make it work is a LOT of work. Keep It Simple, Stupid - that's a usable way to function in the world of Open Source.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

What are the RAM requirements for ZFS? Does it depend on drive size, network speed, etc.? Does ZFS require ECC memory?
 

mtu

Active Member

Reaction score: 111
Messages: 166

What are the RAM requirements for ZFS? Does it depend on drive size, network speed, etc.? Does ZFS require ECC memory?
There used to be a "1GB of RAM for every 1TB of storage" rule-of-thumb floating around, but that's increasingly being disputed. You'll probably be fine even with a little less, just don't enable deduplication unless you know what you're doing, because it eats RAM like crazy.

The ECC memory thing is a stupid piece of misinformation that's stuck for no good reason. Everything a computer does is always at risk from random bitflips in memory. ZFS is at risk as much as any other filesystem, database or whatever.

Saying that "ZFS is unsafe unless you have ECC memory" is like saying "Driving a Hyundai through Wyoming is unsafe, because you might get hit by a meteorite." Well duh, that risk is always present, no matter the vehicle or location.

You can equip your server with ECC memory, and it will be beneficial – same as you can strengthen your car roof against meteorite strikes. But that has nothing to do with ZFS (or Hyundai cars) in particular.
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

I've seen somewhere that you should probably stay away from ZFS if you have less than 2 GB of RAM. Too lazy to dig up a link for that.

No, it does not depend on drive size or network speed. Those are irrelevant to a filesystem.

No, ECC memory is not required. ZFS will run fine (and not care) if your RAM is non-ECC. FWIW, ECC RAM is about 2-3 times more expensive than non-ECC RAM.

I would suggest that OP go read the FreeBSD Handbook... it does contain lots of useful info.
 

gpw928

Aspiring Daemon

Reaction score: 220
Messages: 527

What do you think about HDD Docking Stations from same brand? The USB 3.1 dual-bay is quite expensive.
I did consider multi bay docking stations.
They offer superior stability to my chosen solution of sitting naked disks on top of anti-static packets.
However, docking stations generally have to share the USB bandwidth between the drives.
I wanted full bandwidth to each external disk. That was desirable for a planned re-organisation of the ZFS tank.
The USB disks are usually only deployed occasionally for relatively short durations for backups, so they are not sitting around all the time waiting to get bumped.
To that end, I bought a StarTech PCIe card with two USB 3.1 Gen 2 ports and two matching StarTech adapters.
Then my old motherboard died, and the new motherbard has multiple integrated USB 3.2 Gen 2 ports.
 

grahamperrin

Daemon

Reaction score: 535
Messages: 1,731

There used to be a "1GB of RAM for every 1TB of storage" rule-of-thumb floating around, but that's increasingly being disputed.

+1 to disputing that rule of thumb.


The FreeBSD Handbook is not the best point of reference for ZFS.

<https://docs.freebsd.org/en/books/handbook/zfs/#zfs-term-compression> is wrong.

<https://docs.freebsd.org/en/books/handbook/zfs/#_memory> gives the disputed rule of thumb.

<https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced-tuning-arc_max> whilst vfs.zfs.arc_max is accepted for backwards compatibility, in OpenZFS the variable is vfs.zfs.arc.max … and so on. From <https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075661.html>:

'arc' was converted to a subtree.

<https://cgit.freebsd.org/src/commit/?id=0421f257b20f41e33fb69e067b47beed2c5bd3bd>

FreeBSD: Add legacy arc_min and arc_max
 
Top