ZFS Questions about ZFS on FreeBSD and Linux

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

The FreeBSD Handbook is not the best point of reference for ZFS.
To be honest, I agree. The chapter on ZFS tries to do too much at once. It was a good place to get started, though. I got ideas on what ZFS can do. However, when it comes to getting a handle on how ZFS even works - I had to look beyond the FreeBSD Handbook. Well, that was part of the fun! :p
 

ralphbsz

Son of Beastie

Reaction score: 2,301
Messages: 3,212

What are the RAM requirements for ZFS? Does it depend on drive size, network speed, etc.?
Personal experience: My machine is a 3GiB 32-bit Atom. It has a total of 6 TiB of ZFS pools connected (I think, too lazy to look, that's from memory), one of which uses mirroring. It works flawlessly. BUT: I do not attempt to use compression or dedup.

Does ZFS require ECC memory?
That question was already answered above: no, it does not require it. But the answer is a little bit more complicated. Let me answer it this way: Does a car need airbags? No, not really, it is driveable without airbags. If you don't have seat belts, buying air bags is really pointless. But once you have seat belts, air bags are a really good idea.

Explanation: The greatest risk to data stored on disk in a file system comes from disk errors, both detected ones (where the disk says "I can't read this sector"), and undetected ones (where the disk returns gibberish to you, without telling you). ZFS can detect all these errors, by using CRCs stored separately on disk. And if you have either the space or extra disks, it can even be safe against detected errors. So the most important source of data corruption is gone. What is the next biggest one? Memory errors. That's why with ZFS, having ECC suddenly becomes relatively more important, even if the absolute number of expected memory errors does not increase.
 

grahamperrin

Daemon

Reaction score: 535
Messages: 1,731

… I do not attempt to use compression …

If it's on by default, do you switch it off?



Correction


Having read so much about Zstd (<https://issue.freebsdfoundation.org/publication/?m=33057&i=706299&p=25&ver=html5> and so on) I imagined that Zstd had become the default (i.e. when compression = on). My mistake; the defaults now are still as they were in 2015.

<https://openzfs.github.io/openzfs-docs/man/7/zfsprops.7.html#Native_Properties>
<https://github.com/openzfs/zfs/blob...82d7a9eae90259b15a5a/man/man7/zfsprops.7#L801>
 

mtu

Active Member

Reaction score: 111
Messages: 166

Let me answer it this way: Does a car need airbags? No, not really, it is driveable without airbags. If you don't have seat belts, buying air bags is really pointless. But once you have seat belts, air bags are a really good idea.
Automobile analogies in IT usually go wrong, but I think you've nailed it!
 

fcorbelli

Active Member

Reaction score: 54
Messages: 187

A lot of very strange things.

1) UFS vs ZFS? No no no unless on watch-size machine
2) RAM? 4GB and beyond
3) ECC? There is the infamous "kill-the-root". In a single catastrophic case (of a RAM failure) everything (on the disk!) could be lost by a (evil) resilvering. Yes, very rare. Thoerically. In fact much better to get a good backup (next point)
4) Backups on USB? Go to NTFS, encrypt by zpaq or whatever. Really never-never something like ZFS on external drives for backup
5) Mixing with Windows (aka: samba server). Only turn ON default compression @lz4. Nothing more to do. But be warned: >260 chars path is easy on BSD, not so much with NTFS (yes, some registry hack DO exists, but too dangerous). Rsync and robocopy does not works very well with charsets, even rync-over-nfs or rsync-over-samba. Of course other tools can be used
6) ZFS and Linux. Personally I NEVER try to mount anything on Linux. Sometimes Solaris <=> BSD. BSD 13.0 is, for my very very low trust level, broken on the ZFS implementation. Simply I do NOT use 13 in production, and whenever possible use always 11.x that, in fact, works
7) Encryption. For me the best solution is an encrypter volume (aka: into a file). Then versioned backup of the volume on USB-NTFS-formatted drive, NAS, cloud or whatever. Full filesystem encryption is a ticketing bomb. Of course if you have a bunch of Netflix video not a big deal, but if you have some insubstitutible data it's a different story
8) Deduplication. It is almost useless in any scenarios, precious for reducing wear extracting backup on SSDs to verify. In this case a couple of the cheapest SATA SSD in stripe ("raid-0") are mandatory. Please note: with deduplication a "fsck" DO exists in BSD, this is a very good reason to DO NOT use deduplication (only for temporary storage)
 

mer

Well-Known Member

Reaction score: 290
Messages: 488

I'm not discounting fcorbelli statement "...13.0 is, for my very very low trust level, broken on the ZFS implementation. …"; but would like to point out my experience/opinion:

I've had a couple of machines using ZFS for quite a while. One has a mirror pair (used for home directories) that actually started out as a singleton way back in FreeBSD-9 and has been migrated system upgrades and new hardware and making it mirrored. Currently running FreeBSD-13.0-RELEASE on it and have not seen any issues. It's relatively bog-standard ZFS use, pretty much default properties, worst thing is "NFS exporting". The boot on that system is a mirrored pair that was from probably a fresh 12.x install, then freebsd-updated to 13.0-RELEASE.
Another system was 12.x, freebsd-updated to 13.0-RELEASE, single internal M2 device.

Following is my opinion:
I have not seen any issue in my normal use cases that would support the "...13.0 is broken on the ZFS implementation".

If you go back through the mailing lists, there was a lot (really a lot) of discussion about switching ZFS implementations and lots of call for testers and lots of back and forth getting things fixed. Is it possible that some corner cases were missed? Absolutely possible, but overall a lot of effort was put into getting it right.

Areas of concern are always:
Compatibility. Different ZFS implementations always had the potential for adding features/flags that conflict. If you don't use them, it's not an issue. Think before doing "zpool upgrade"

Bootloader. Always an area of concern if you are using Boot Environments. What if the gptzfsboot or efi bootloader code is wrong and can't recognize my pools? Well, that's happened in the past and yes, it is frustrating, but you often aren't alone. Searching the forum here has a few threads about it and how to fix it. Typically, answers are tied into "be careful if you zpool upgrade things that affect boot environments".

In the end I lament "dropping a FreeBSD implementation for a Linux one", but in this case the switch was "small interest group, pushing patches upstream, seeing them not get incorporated" for "upstream rebased to a larger, more vibrant and active group of users, so FreeBSD should rebase to take advantage".

Again, I am not discounting or diminishing anyone's personal experience, but I too would like to know what is really broken on the current ZFS implementation (vs what acts differently).
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

6) ZFS and Linux. Personally I NEVER try to mount anything on Linux. Sometimes Solaris <=> BSD. BSD 13.0 is, for my very very low trust level, broken on the ZFS implementation. Simply I do NOT use 13 in production, and whenever possible use always 11.x that, in fact, works
7) Encryption. For me the best solution is an encrypter volume (aka: into a file). Then versioned backup of the volume on USB-NTFS-formatted drive, NAS, cloud or whatever. Full filesystem encryption is a ticketing bomb. Of course if you have a bunch of Netflix video not a big deal, but if you have some insubstitutible data it's a different story
7) So you use ZFS without encryption feature, because it must not be OpenZFS but Solaris ZFS.
8) What encryption volume do you suggest?
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

7) So you use ZFS without encryption feature, because it must not be OpenZFS but Solaris ZFS.
8) What encryption volume do you suggest?
7) I would expect Solaris ZFS to have some kind of support for encryption, even if it's not obvious right away. I did not read the docs, this is just an educated guess.

8) What do you mean by 'encryption volume'?
You can have an 'encryption method'. Also, ZFS does not have 'volumes', but 'datasets'. 'volumes' is an NTFS thing. I would expect OpenZFS to support a few ways to encrypt datasets (if compiled with appropriate options).
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

8) What do you mean by 'encryption volume'?
You can have an 'encryption method'. Also, ZFS does not have 'volumes', but 'datasets'. 'volumes' is an NTFS thing. I would expect OpenZFS to support a few ways to encrypt datasets (if compiled with appropriate options).
Provide some command line examples about how to do that.
 

grahamperrin

Daemon

Reaction score: 535
Messages: 1,731

… Solaris ZFS … encryption, …

Yes and no, loosely speaking. IIRC:
  1. Oracle's implementation (post-Sun) was closed source
  2. source code was leaked but necessarily avoided
  3. there was work towards a compatible open source implementation
  4. ultimately we gained a superior open source implementation that did not aim for compatibility with Oracle's.
That's a highly condensed short story. I might be hung, drawn and quartered for errors and omissions. I should at least give credit to the person who was largely responsible for (3) … working from memory I can't be certain but the initials J.L. come to mind.

Postscripts

J.L. = Jorgen Lundman, <https://github.com/lundman>, IIRC he was pivotal (correct me, please, if I'm wrong).

In the Wayback Machine: What will happen to ZFS Crypto? - illumos.org (2010-08-24)

Oracle Solaris 11 Kernel Source-Code Leaked - Phoronix (2011-12-19)

From Disgruntled employee? Oracle doesn’t seem to care about Solaris 11 code leak | Ars Technica (2011-12-21):

… That proprietary code includes the source for Solaris' kernel-level cryptographic framework daemon, logical link control driver, and code for mounting NFS filesystems. A significant portion of the code in "closed," however, also carried the CDDL header; it's not clear if Oracle intended to make this previously open code closed or not.

The code drop is being given a wide berth by developers associated with two of the open development projects that have forked from OpenSolaris, the open-source kernel effort that contributed much of the new technology in Solaris 11. …

… Bootloader …

I don't have the link handy, but there's work in progress towards – at least – better documentation. In the meantime:
  1. <https://lists.freebsd.org/archives/freebsd-current/2021-July/000331.html> from Warner Losh
  2. <https://lists.freebsd.org/archives/freebsd-current/2021-July/000342.html> concerning a typo
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

In the end I lament "dropping a FreeBSD implementation for a Linux one", but in this case the switch was "small interest group, pushing patches upstream, seeing them not get incorporated" for "upstream rebased to a larger, more vibrant and active group of users, so FreeBSD should rebase to take advantage".

Again, I am not discounting or diminishing anyone's personal experience, but I too would like to know what is really broken on the current ZFS implementation (vs what acts differently).
Probably he doesn't like that Linux supports ZFS now. Let's be clear, FreeBSD even didn't have any copyright ownership on ZFS to begin with, it all belongs to Oracle, so don't pretend FreeBSD devs invented it. The licensing issues are the first thing to be taken care of, which looks like FreeBSD guys don't care about, this is why they end in this mess.
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

Yes and no, loosely speaking. IIRC:
  1. Oracle's implementation (post-Sun) was closed source
  2. source code was leaked but necessarily avoided
  3. there was work towards a compatible open source implementation
  4. ultimately we gained a superior open source implementation that did not aim for compatibility with Oracle's.
That's a highly condensed short story. I might be hung, drawn and quartered for errors and omissions. I should at least give credit to the person who was largely responsible for (3) … working from memory I can't be certain but the initials J.L. come to mind.

Postscripts

J.L. = Jorgen Lundman, <https://github.com/lundman>, IIRC he was pivotal (correct me, please, if I'm wrong).

In the Wayback Machine: What will happen to ZFS Crypto? - illumos.org (2010-08-24)

Oracle Solaris 11 Kernel Source-Code Leaked - Phoronix (2011-12-19)

From Disgruntled employee? Oracle doesn’t seem to care about Solaris 11 code leak | Ars Technica (2011-12-21):





I don't have the link handy, but there's work in progress towards – at least – better documentation. In the meantime:
  1. <https://lists.freebsd.org/archives/freebsd-current/2021-July/000331.html> from Warner Losh
  2. <https://lists.freebsd.org/archives/freebsd-current/2021-July/000342.html> concerning a typo
Pretty amazing research... That's what makes FreeBSD forums so interesting. Yeah, sometimes people get bashed for not understanding stuff correctly (I was on the receiving end of that in a couple threads from shkhln ).
 

mer

Well-Known Member

Reaction score: 290
Messages: 488

Let's be clear, FreeBSD even didn't have any copyright ownership on ZFS to begin with, it all belongs to Oracle, so don't pretend FreeBSD devs invented it.
Never once did I say that FreeBSD devs invented or even owned ZFS. Sun Microsystems is the original copyright holder, to my understanding it has always been CDDL license; same license as Java. Oracle changed licensing after they bought Sun; that only changes the licensing for versions after that date. Oracle is the one that changed the licensing on Java, not Sun.

Linux kernel actually tries it's best to not support ZFS. Plenty of information in the Linux mailing lists; currently ZFS is treated as an external module, a "non-GPL" binary if you will. There were one or two situations where the Linux kernel changed the exposure of something in the VFS layers that made ZFS not work. ZoL folks worked around it to use the interfaces made available.

The licensing issues are the first thing to be taken care of, which looks like FreeBSD guys don't care about, this is why they end in this mess.
All I'm saying on this is I believe you are completely wrong and you need to go back and read the freebsd arch and hackers mailing lists, go back to about FreeBSD-9.0 time and then work your way to present day.
As for "...in this mess" what mess are you talking about? There is no mess, ZFS, starting as OpenZFS eventually had forks, just like every other Open Source project out there (how many derivatives of FVWM are there?). One of the forks simply started being more actively developed for features and bugs, so OpenZFS rebased to it. FreeBSD and users of the other fork had a choice: follow that rebase or become the only users of a fork. Following the rebase is the better long term solution because bugs tend to get fixed faster, features added faster, easier in the long run to maintain.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

Linux kernel actually tries it's best to not support ZFS. Plenty of information in the Linux mailing lists; currently ZFS is treated as an external module, a "non-GPL" binary if you will. There were one or two situations where the Linux kernel changed the exposure of something in the VFS layers that made ZFS not work. ZoL folks worked around it to use the interfaces made available.
Linus doesn't want ZFS code inside the Linux kernel code itself, because they can't claim ownership on the code. Rhis is why they are enhancing BTRFS. However, you can use it as a module, like Nvidia proprietary drivers. Whether it's easy to do due to syscalls and APIs is a different story.
 

hardworkingnewbie

Active Member

Reaction score: 151
Messages: 153

Linus doesn't want ZFS code inside the Linux kernel code itself, because they can't claim ownership on the code. Rhis is why they are enhancing BTRFS. However, you can use it as a module, like Nvidia proprietary drivers. Whether it's easy to do due to syscalls and APIs is a different story.
It's much simpler: the Linux kernel is under the GPLv2 license, and probably will be for a much longer time since Linus Torvalds does not want to make a switch to GPLv3. The GPLv2 license is incompatible with CDDL. Since they didn't want a law suit by Sun, now Oracle, ZFS will never be shipped in the main line kernel until this issue has been resolved somehow.

Linux has its own COW file system builtin, also originally and ironically sponsored by Oracle, called Btrfs. This abomination was merged into the Linux kernel back in 2009 and under development since then. Today the question most asked by potential users is still "Is it stable yet?" Basically it's slow and garbage, if you really want to have the more advanced features and will eat your data when you least expect it. ZFS has also much more features, which Btrfs probably will never see. Many major Linux distributions just have given up on Btrfs totally.

There's also some developer named Kent Overstreet on his own interpretation of how a COW file system under Linux should look like, which he called Bcachefs. It's based great lengths on Bcache, which also is Overstreet's brain child. So far he attempted to get his baby merged into the kernel once, but they told him to rewrite much of his stuff and he went back into his developer cave, busy working on that.

So to come to a conclusion: if you are on Linux and want a stable COW file system, then ZFS is your only choice. Period.
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

It's much simpler: the Linux kernel is under the GPLv2 license, and probably will be for a much longer time since Linus Torvalds does not want to make a switch to GPLv3. The GPLv2 license is incompatible with CDDL. Since they didn't want a law suit by Sun, now Oracle, ZFS will never be shipped in the main line kernel until this issue has been resolved somehow.

Linux has its own COW file system builtin, also originally and ironically sponsored by Oracle, called Btrfs. This abomination was merged into the Linux kernel back in 2009 and under development since then. Today the question most asked by potential users is still "Is it stable yet?" Basically it's slow and garbage, if you really want to have the more advanced features and will eat your data when you least expect it. ZFS has also much more features, which Btrfs probably will never see. Many major Linux distributions just have given up on Btrfs totally.

There's also some developer named Kent Overstreet on his own interpretation of how a COW file system under Linux should look like, which he called Bcachefs. It's based great lengths on Bcache, which also is Overstreet's brain child. So far he attempted to get his baby merged into the kernel once, but they told him to rewrite much of his stuff and he went back into his developer cave, busy working on that.

So to come to a conclusion: if you are on Linux and want a stable COW file system, then ZFS is your only choice. Period.
Sounds like you did your homework, but some links would be nice.
 

Menelkir

Active Member

Reaction score: 302
Messages: 233

Basically it's slow and garbage
Actually no. BTRFS have an ok performance and a good footprint of cpu/ram usage. The problem is that isn't reliable at all, and I'm not talking just about RAID5/6, I see scenarios that was unacceptable to production, at least for me (e.g mirrors that wasn't "mirrored" properly, borked metadata mirror with an idle machine, and the list goes on).
 

bsduck

Active Member

Reaction score: 161
Messages: 212

Actually no. BTRFS have an ok performance and a good footprint of cpu/ram usage.
I agree. And when used as a basic filesystem (I mean not taking advantage of advanced features and using it like you would use ext4 or UFS2) it's even one of the fastest, at least on spinning HDDs.

The problem is that isn't reliable at all
I agree too. Bugs like this one is not what I expect from a reliable filesystem: https://bugs.archlinux.org/task/63733
 

astyle

Aspiring Daemon

Reaction score: 376
Messages: 858

Links to what?
Well, you told an interesting story, but where did you get that from? If there's a Wikipedia article about that, link to that... If there was an announcement on the project page, link to that. Links give credibility to the stories shared.
 
OP
B

bugzeo

Member

Reaction score: 18
Messages: 95

Well, you told an interesting story, but where did you get that from? If there's a Wikipedia article about that, link to that... If there was an announcement on the project page, link to that. Links give credibility to the stories shared.
I've heard somewhere that BTRFS performance can drastically degrade if you have an awful amount of snapshots. I think they recommend having less than 100. I don't know if ZFS can do that.
 
Top