ZFS on Linux

Personally I would place boot&kernel on ext2 or ext4.
And place data on zfs, because of the easy of backup & snapshot
For /,root i would be open to any option, ext4,zfs,xfs,jfs,...
Note, a bad internet connection never give a good "quality of service". That looks like a bad choice.
[My limited take on this problem]
 
_martin said:
My 2c: this is way too specific question and requires background information about the $job and what task it's going to do. Is it actually being shipped (pun intended) on ships at sea? If so I'd say simpler the better. I have no idea what is the upgrade strategy on those devices. From personal experience many specific devices are usually set and forget.

Are those only VMs or physical HW? I'd stick to ext4 in either case probably. Given you have robust recovery solution at place (such as rear,etc.).
Click to expand...
proxomox as you may have read, does not officially support EXT4, mdamd, etc. Only ZFS. This is an important point.

Now, yep the new hardware will be shipped to the ships by a smaller ship containing maybe some sheep hahaha kidding. Yep. New hardware running proxmox.
The original setup was based on bare metal. Then came some virtualization solution of which I was kept ... outside any decision (dangerous to leave the DBA out of all this), now we are thinking of dumping this in favor of our own virtualization solution based on proxmox.

As I wrote EXT4 is fine for me, never failed me (in contrary to XFS), but proxmox says this is not supported. + there is no comparison between the two FS's really.
 
Eric A. Borisch said:
ZFS works well on linux; sometimes the packaging (getting kmods updated for the latest kernel) trips up smooth upgrades, but that’s the worst I’ve encountered in many years of use.

If you need to run Linux, and you’re comfortable with (and would like the benefits of) ZFS, there’s no reason to avoid it.
Click to expand...

Great point.
 
Recovery. I think that is a key requirement.
Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.
 
mer said:
Recovery. I think that is a key requirement.
Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.
Click to expand...
great point, now I get what Boot Environment means.
 
  • Like
Reactions: mer
achix said:
our current images run since 2005... unpatched , postgresql 8.4, I dont even remember debian versions or kernels. Those are not connected to the net, so the no1 aim is to just work. No corruption. Recovery is not so cheap in terms of time. Anyways, all those aspects are subject to redesign. We are in the middle of this huge upgrade.
Click to expand...
mer said:
Recovery. I think that is a key requirement.
Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.
achix said:
great point, now I get what Boot Environment means.
Click to expand...
Click to expand...
Perhaps have a closer look at Boot Environments and especially their relation to recovery; for example Managing Boot Environments and Let’s Talk OpenZFS Snapshots; also have a look at vermaden's message. For a deeper ZFS dive I recommend the two ZFS books: FreeBSD Development: Books, Papers, Slides

Boot Environments (BEs) are a feature enabled by ZFS. From an implementation standpoint they are "almost free of charge"; ZFS does the heavy lifting. BEs are (very) useful for managing updates: patches, minor and major version updates of FreeBSD and software packages. Those updates are related to system environments and system changes, hence the name Boot Environments: if an update does not function as intended (extreme case: does not boot) you can easily go back to a previous Boot Environment: the system as it was before the update. BEs are not intended for recovery in case of corrupt (user) data. Currently there are no Boot Environment options for Linux.

When there is data corruption to such an extent that ZFS cannot correct the errors anymore and you have lost a ZFS pool, there are basically no tools available to recover a failed pool. That means all data in that pool is lost and you'll have to resort to backups. This is a kind of a consequence of ZFS taking very good care of your data (depending on the chosen redundancy setup) but if, in the end, there is a non-recoverable error (even in just one file), you will have lost all data in the pool where that file belongs to.

As you have described your situation so far the environment is an all Linux environment, so please be aware of what properties and additional tools mentioned belong exclusively to a FreeBSD ZFS environment and consider SirDice's suggestion*.

___
* unless you are in a position to consider a switch from Linux to FreeBSD: no proxmox on FreeBSD but, we do have jails as a lightweight OS-level virtualisation mechanism :)
 
I don't put the linux-bootloader on zfs.
I don't put the linux-kernel on zfs.
It's just too dangerous.
This means i use multiple file-systems.
 
achix said:
Any opinion on the Linus Torvalds stmt back in 2020 : "do not use ZFS"?
Click to expand...
Yes, Linus threw a tantrum 'cause the Openzfs guys wouldn't give him code under terms he likes so he decided to spread FUD about ZFS.

The statement "(ZFS) was always more of a buzzword than anything else..." is so patently and obviously absurd that I can't believe someone as smart as Linus uttered it in good faith. ZFS is more than a filesystem. It's a volume manager and software RAID layer, and it makes the Linux md and LVM crapola look primitive.

Here's a longer take on the matter:
arstechnica.com

Linus Torvalds says “Don’t use ZFS”—but doesn’t seem to understand it

Linus should avoid authoritative statements about projects he's unfamiliar with.
arstechnica.com arstechnica.com
 
For my linux machine, I've been running root on zfs for a while without any issues on Debian. I originally tried to also boot from zfs, but grub's zfs support is sorely painful that I moved the /boot to an ext3 partition.
 
Erichans said:
Perhaps have a closer look at Boot Environments and especially their relation to recovery; for example Managing Boot Environments and Let’s Talk OpenZFS Snapshots; also have a look at vermaden's message. For a deeper ZFS dive I recommend the two ZFS books: FreeBSD Development: Books, Papers, Slides

Boot Environments (BEs) are a feature enabled by ZFS. From an implementation standpoint they are "almost free of charge"; ZFS does the heavy lifting. BEs are (very) useful for managing updates: patches, minor and major version updates of FreeBSD and software packages. Those updates are related to system environments and system changes, hence the name Boot Environments: if an update does not function as intended (extreme case: does not boot) you can easily go back to a previous Boot Environment: the system as is was before the update. BEs are not intended for recovery in case of corrupt (user) data. Currently there are no Boot Environment options for Linux.

When there is data corruption to such an extend that ZFS cannot correct the errors anymore and you have lost a ZFS pool, there are basically no tools available to recover a failed pool. That means all data in that pool is lost and you'll have to resort to backups. This is a kind of a consequence of ZFS taking very good care of your data (depending on the chosen redundancy setup) but if, in the end, there is a non-recoverable error (even in just one file), you will have lost all data in the pool where that file belongs to.

As you have described your situation so far the environment is an all Linux environment, so please be aware of what properties and additional tools mentioned belong exclusively to a FreeBSD ZFS environment and consider SirDice's suggestion*.

___
* unless you are in a position to consider a switch from Linux to FreeBSD: no proxmox on FreeBSD but, we do have jails as a lightweight OS-level virtualisation mechanism :)
Click to expand...

Again great insight and real no-BS help which justifies why I came here and not in a linux place. I have done some things with vm/bhyve and jails of course. However distribution of new images in jails are kinda hard/not a thing, + all our ship/vessel system is based on Linux and some archaic tools that we still have to run (uucp). Yep uucp, a tool that was obsoleted in my school when I was a freshman back in 1986, go figure.
 
For what it is worth, we use proxmox at $DAYJOB. I have not had much to do with it, even if we have several virtual servers running in it. Developemt servers need a snapshot/rollback from time to time. And I had not much contact with it because it simply works without a glitch up to now. Maybe it's only me, but they look like they know their stuff.
 
achix said:
Any opinion on the Linus Torvalds stmt back in 2020 : "do not use ZFS"?
Click to expand...
Torvalds was speaking about the license issues this might cause. If it really comes to the quality of file systems he is not the guy to listen to, because file system development was never done by him. I would then have a look at Dave Chinner (XFS in the Linux kernel) or Theodore T'so (father of ext2/3/4).

By the way the default setup which Proxmox uses is ext4+LVM (have a look at this: https://pve.proxmox.com/wiki/Installation), not ZFS. You can choose ZFS though. LVM snapshots are known to work, but to be slow as well, so if you really want snapshots obviously ZFS is the way to go.
 
Proxmox-Wiki-Installation:
Proxmox VE can be installed on ZFS. As ZFS offers several software RAID levels, this is an option for systems that don’t have a hardware RAID controller. The target disks must be selected in the Options dialog. More ZFS specific settings can be changed under Advanced Options (see below).

[Warning] ZFS on top of any hardware RAID is not supported and can result in data loss.
Click to expand...
ZFS offers functionality and data protection far beyond that of any hardware RAID. (ZFS on top of hardware RAID is a definite no.)

An additional hardware RAID consideration to keep in mind when not relying on ZFS. Current hardware RAID controllers are likely not the hardware RAID controllers you'd want or expect. Hardware RAID controllers of the past in their server hardware RAID controller incarnation had real protection mechanisms against some data corruption but the landscape has changed: Hardware Raid is Dead and is a Bad Idea in 2022
 
Erichans said:
Hardware RAID controllers of the past in their server hardware RAID controller incarnation had real protection mechanisms against some data corruption but the landscape has changed: Hardware Raid is Dead and is a Bad Idea in 2022
Click to expand...
This is good to know. A good rule of thumb in the past was that if your hardware RAID controller cost lest than $1000, you probably shouldn't use it. Oh, you're feeling lucky? Data corruption is your door prize. Yup, all your backups for the past month are corrupt too. Congratulations!
 
hardworkingnewbie said:
Torvalds was speaking about the license issues this might cause. If it really comes to the quality of file systems he is not the guy to listen to, because file system development was never done by him. I would then have a look at Dave Chinner (XFS in the Linux kernel) or Theodore T'so (father of ext2/3/4).

By the way the default setup which Proxmox uses is ext4+LVM (have a look at this: https://pve.proxmox.com/wiki/Installation), not ZFS. You can choose ZFS though. LVM snapshots are known to work, but to be slow as well, so if you really want snapshots obviously ZFS is the way to go.
Click to expand...
thnx
no, they support only ZFS : https://forum.proxmox.com/threads/z...raid-stability-and-reliability-of-zfs.116871/
 
tweet on UBUNTU 21.10 about Ubuntu zfs-linux package--bug 1906476

PSA: DO NOT UPGRADE TO UBUNTU 21.10 IF YOU USE ZFS. DO NOT create a new installation of Ubuntu 21.10 with ZFS either.

It *will* corrupt your filesystem irreparably. There’s a warning in the release notes – but only after 2800 words >_>
Click to expand...
Erichans said:
As you have described your situation so far the environment is an all Linux environment, [...]
Click to expand...
This is now in the past (Ubuntu 21.10 has reached EoL); I hope and expect that Ubuntu has learned from this and that this is a once-in-a-lifetime thing. However, it shows the possibility of problems that can arise when the ZFS part cannot be developed in sync with the kernel of an OS and the responsibility of integrating/matching ZFS with a current kernel is shifted to the team that constructs a specific OS distribution.
 
achix said:
thnx
no, they support only ZFS : https://forum.proxmox.com/threads/z...raid-stability-and-reliability-of-zfs.116871/
Click to expand...
Wrong. As you can see here in the most recent iteration of their own documentation it says this below:


After selecting Install Proxmox VE and accepting the EULA, the prompt to select the target hard disk(s) will appear. The Options button opens the dialog to select the target file system.

The default file system is ext4. The Logical Volume Manager (LVM) is used when ext4 or xfs is selected. Additional options to restrict LVM space can also be set (see below).
 
Alain De Vos said:
I dual boot freesd-on-zfs , gentoo-linux-on-ext4.
When i boot gentoo-linux i mount the freebsd-zpools & this works fine and perfect.
Click to expand...
Fantastic, I used to think that it's better not to do this just in case due to compatibility issues ) Let's say when upgrading the system, we sometimes have to do a zpool upgrade. How about this in this case?

Would that zpool be created on FreeBSD? And how: just zpool create mypool or with some special zpool properties used?
 
Alain De Vos said:
but even on this freebsd-desktop on which i write this message the bootloader is on ufs & root filesystem is on zfs. Having the bootloader on a simple filesystem (ufs) makes it more robust I think. This is my personal take but mileage may vary.
Click to expand...
What if I use zfsboot to root-on-zfs on FreeBSD? I just set up the system like this:
Code: 
zpool create -m none mypool /dev/diskid/....
zfs create -o mountpoint=/mnt mypool/rootfs
cd /mnt
wget https://download.freebsd.org/ftp/releases/amd64/13.0-RELEASE/base.txz
wget https://download.freebsd.org/ftp/releases/amd64/13.0-RELEASE/kernel.txz
[...]
zpool set bootfs=mypool/rootfs zpool
[...]

# install zfsboot
dd if=/boot/zfsboot of=/dev/diskid/.... count=1
dd if=/boot/zfsboot of=/dev/diskid/.... iseek=1 oseek=1024

# and in /boot/loader.conf
vfs.root.mountfrom="zfs:mypool/rootfs"
zfs_load="YES"

So I don't have partitions at all. Why is this way worse? Less stable system? Need to use Legacy BIOS? (zfsboot - bootcode for ZFS on BIOS-based computers). And the result is slightly lower performance? Theoretically, I can also import this zpool when booting to Gentoo.

It's just that the zfsboot method allows me to put the entire disk under ZFS management (although this is not currently necessary). But there is another aspect that this method is simple and beautiful: we use only ZFS and it manages the entire disk, and /etc/fstab can be either completely empty or intended for various small file systems like tmpfs. What do you think about zfsboot?
 
You must log in or register to reply here.
Back
Top