ZFS ZFS FAQ

My experience is when ZFS detects an incomplete transaction, it rolls back. Example: I go through multiple edit + debug iterations not realizing my battery is dangerously low. Laptop dies. I restore power and reboot. My last few edits are gone.

I did not have to wait hours using fsck to detect an error and figure out how to fix it.

The point is ZFS gave me a consistent file system. The data loss is on me, not ZFS.
 
ralphbsz said:
What happens if it detects an error? You know that you have lost data. I think that's better than the data being silently corrupted, and you getting wrong data. It is also better than your file system crashing or acting weird, if the error was in metadata.
Click to expand...

ZFS has a nice feature called "ditto" blocks. It actually keeps 3 copies of "filesystem internal metadata" - global metadata that isn't filesystem specific) (and if possible spreads them out over different vdevs (disks). It also keeps 2 copies of per-filesystem metadata. And everything checksummed of course. Ie, the chance of unrecoverable metadata corruption is pretty low. However, as always - if the single drive dies then everything goes. This is the bad thing about SSDs I think - they often just die completely - unlike spinning rust that often dies more "nicely"...
 
Alain De Vos said:
I use ZFS on my desktop for all data I can loose. :) In my case on /usr/ports & /usr/src .
On one disk backup is advised, as there is no fsck, and if you run into a ZFS bug or meta-data corruption you have problems.
Click to expand...

Actually, in all my years (since it first came to life in the Solaris days) of running ZFS on both big and small servers I've never had a metadata corruption that I couldn't recover from.

The most severe one I ran into was a problem with a Solaris kernel bug where if the server had a memory problem and/or rebooted due to power outage at the wrong time could cause corruption in the ZFS spacemap data structure (manages free space) on disk once - which caused a Solaris kernel crash when attempting to write to the filesystem. The way we fixed that was to import the zpool readonly and copy all the files to a new disk. A nuisance but... Probably could have fixed it by patching the spacemap via zdb too but ... Anyway that was a long time ago (in the Solaris 10 days) and that bug is long fixed.
 
Alain De Vos said:
On one disk backup is advised, as there is no fsck, and if you run into a ZFS bug or meta-data corruption you have problems.
Click to expand...
Agreed with the advice, wondering about the reasoning. Tools like fsck are necessary for filesystems that don't do checksumming and the like, and once you actually need them, you are already fsck'd ;) To force a check of your zfs, scrub it. That's more comfortable than fsck, as it runs in the background while the fs can still be used.

Yes, once zfs finds errors it can't repair (which of course is most likely on a single-disk pool), you're basically in the same situation as with any fsck that is unable to repair the mess it found. Another shortcoming of the typical fsck approach is of course: It might not find anything wrong, and still you have corrupted files. You'll only ever notice once you actually need the contents of these files.
 
for a single disk/pool laptop ..


What I did was .. create a 240 gig zvol on my main storage machine.
Then install the laptop with a 240gig pool.

Write a quick script to connect to the server and compare the timestamps of the zvol/pools.. if the server is newer initiate a receive to the laptop.. if its older. do a send from the laptop. This basically allows you to replicate the pool and update your laptop every time you use it ...

so when you go to the coffee shop and write that new song .. as soon as you get home its automatically replicated to your main pool.. and if your working on your desktop and send a bunch of new music to your laptop pool.. the next time you connect it will pull it all down.

just don’t forget to exclude your tmp and swap :)


if the laptop ever dies.. who cares, install freebsd and import the pool ..

everytime the data is sent as well, it will check for errors.. so the chances of getting corruption is basically 0.
 
Alain De Vos said:
I use ZFS on my desktop for all data I can loose. :) In my case on /usr/ports & /usr/src .
On one disk backup is advised, as there is no fsck, and if you run into a ZFS bug or meta-data corruption you have problems.
Click to expand...

I am living on ZFS for all my present installations -
  1. Server as a BHYVE hypervisor with raidz (running on Hetzner SX62);
  2. several BHYVE guests on the server (all ZFS);
  3. desktop system with AMD GPU, MATE, KDE, XFCE and desktop applications (ZFS mirror with SSD ZIL and L2ARC);
  4. laptop system with a single HDD and Intel GPU.
Everything seems to run smoothly so far... :)

P.S. The server hosting provider Hetzner does not advertise FreeBSD on their website, but in fact it works ;)
 
I use ZFS on my desktop for all data I can loose[...]
Click to expand...
I ran into a ZFS "bug" once in two decades (OpenSolaris & FreeBSD), and that was because I took the freedom to tweak an option documented not to be tweaked from the default value...
 
mjollnir said:
I ran into a ZFS "bug" once in two decades (OpenSolaris & FreeBSD), and that was because I took the freedom to tweak an option documented not to be tweaked from the default value...
Click to expand...

That is a good point for FAQ - how to tune ZFS for different applications.
There are different guides in the internet (like this ZFS tuning), but it is a complicated process for an average user. There are many parameters and IMHO insufficient description how to do the practical tuning for a specified purpose and what are the trade-offs in any such case.
 
Hello world :) It would be nice if someone experienced could compare RAID-0+ZFS vs ZFS-STRIPE:
  1. What would be the performance of hardware RAID0 vs ZFS-STRIPE. I did reinstall from RAID0+UFS to ZFS-STRIPE and it seems ZFS is much slower (especially for small files). Not sure how it would work with RAID0+ZFS.
  2. Can I easily add new drives to existing ZFS-STRIPE expanding the pool? It seems impossible with RAID0 without re-creating is from scratch. If possible to expand existing ZFS-STRIPE pool then lower speed is acceptable cost.
Thanks :)
 
zader said:
… see https://www.freebsd.org/doc/handbook/zfs-advanced.html …
Click to expand...

zader would you like to update the link?

<https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced>

phalange said:
… older laptop's (for me, T430s and T61s) …
Click to expand...

Ages?

Dirt-cheap L2ARC can have a great effect on performance in some cases.

Argentum said:
… desktop system with AMD GPU, MATE, KDE, XFCE and desktop applications (ZFS mirror with SSD ZIL and L2ARC); …
Click to expand...

I have a circa 2013 HP EliteBook 8570p with <https://bsd-hardware.info/?probe=646148fc25> three old USB flash drives as cache devices at the time of writing:
  • Kingston DataTraveler 3.0 (29 G) for the boot disk (7,200 RPM HGST HTS721010A9E630)
  • Kingston DataTraveler 3.0 (14 G) for a mobile hard disk drive on USB 2.0 that's primarily for VirtualBox data
  • generic Duracell-branded (15 G) also for the boot disk – this was not planned, but one day I forgot to put the 29 G device in my bag, so I simply added a second cache device (and I'm no rush to remove it from the pool).
Stats etc. from around a week ago:

Code: 
% zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Sep 12 16:24:36 2021
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      0
        Free on Write:                          0
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         43.17   GiB
        Decompressed Data Size:                 92.27   GiB
        Compression Factor:                     2.14
        Header Size:                    0.17%   163.67  MiB

L2 ARC Breakdown:                               81.78   k
        Hit Ratio:                      76.39%  62.47   k
        Miss Ratio:                     23.61%  19.31   k
        Feeds:                                  1.01    k

L2 ARC Writes:
        Writes Sent: (FAULTED)                          195
          Done Ratio:                   99.49%  194
          Error Ratio:                  0.00%   0

------------------------------------------------------------------------

% uptime ; zfs-stats -L
 4:26p.m.  up 19 mins, 5 users, load averages: 0.51, 0.44, 0.32

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Sep 12 16:26:18 2021
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      0
        Free on Write:                          0
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         43.16   GiB
        Decompressed Data Size:                 92.22   GiB
        Compression Factor:                     2.14
        Header Size:                    0.17%   163.04  MiB

L2 ARC Breakdown:                               89.13   k
        Hit Ratio:                      70.23%  62.59   k
        Miss Ratio:                     29.77%  26.54   k
        Feeds:                                  1.09    k

L2 ARC Writes:
        Writes Sent: (FAULTED)                          271
          Done Ratio:                   99.63%  270
          Error Ratio:                  0.00%   0

------------------------------------------------------------------------

% su -
Password:
root@mowa219-gjp4-8570p-freebsd:~ # poudriere ports -u
[00:00:00] Updating portstree "default" with git+https... done
root@mowa219-gjp4-8570p-freebsd:~ # git -C /usr/doc pull --ff-only && git -C /usr/ports pull --ff-only && git -C /usr/src pull --ff-only
Already up to date.
Already up to date.
Already up to date.
root@mowa219-gjp4-8570p-freebsd:~ # exit
logout
% zpool iostat -v
                         capacity     operations     bandwidth
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               428G  36.4G      7      0  1.54M  6.45K
  gpt/Transcend         428G  36.4G      7      0  1.54M  6.45K
cache                      -      -      -      -      -      -
  gpt/cache-transcend  14.4G  19.2M      0      9  2.57K  1.08M
---------------------  -----  -----  -----  -----  -----  -----
august                  235G   677G      8      6   219K   115K
  ada0p3.eli            235G   677G      8      6   219K   115K
cache                      -      -      -      -      -      -
  gpt/cache-august     28.8G  57.0M     53      0   728K  41.4K
---------------------  -----  -----  -----  -----  -----  -----
% tail -f -n 0 /var/log/messages
Sep 12 16:28:16 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): READ(10). CDB: 28 00 2a 37 21 88 00 00 c0 00
Sep 12 16:28:16 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
Sep 12 16:28:16 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): Retrying command, 3 more tries remain
Sep 12 16:28:17 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): READ(10). CDB: 28 00 2a 37 21 88 00 00 c0 00
Sep 12 16:28:17 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): CAM status: CCB request completed with an error
Sep 12 16:28:17 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): Retrying command, 2 more tries remain
Sep 12 16:28:58 mowa219-gjp4-8570p-freebsd kernel: GEOM: da0: the secondary GPT table is corrupt or invalid.
Sep 12 16:28:58 mowa219-gjp4-8570p-freebsd kernel: GEOM: da0: using the primary only -- recovery suggested.
Sep 12 16:28:58 mowa219-gjp4-8570p-freebsd kernel: GEOM: iso9660/GHOSTBSD: the secondary GPT table is corrupt or invalid.
Sep 12 16:28:58 mowa219-gjp4-8570p-freebsd kernel: GEOM: iso9660/GHOSTBSD: using the primary only -- recovery suggested.
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: ugen1.3: <Kingston DataTraveler 3.0> at usbus1 (disconnected)
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: umass0: at uhub4, port 2, addr 3 (disconnected)
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: da0: <Kingston DataTraveler 3.0 PMAP>  s/n 08606E6B6446BFB138159554 detached
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Sep 12 16:29:06 mowa219-gjp4-8570p-freebsd kernel: umass0: detached
^C
% zpool status
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:32:24 with 0 errors on Fri Aug 27 03:51:45 2021
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          gpt/Transcend        ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors

  pool: august
 state: ONLINE
  scan: scrub repaired 0B in 00:53:41 with 0 errors on Fri Aug 27 01:13:10 2021
config:

        NAME                STATE     READ WRITE CKSUM
        august              ONLINE       0     0     0
          ada0p3.eli        ONLINE       0     0     0
        cache
          gpt/cache-august  ONLINE       0     0     0

errors: No known data errors
% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:121 932G GPT                                               - -
  ada0p1         0:123 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:125  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:68   16G freebsd-swap                                      - SWAP
  ada0p3         0:127 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:135 915G zfs                                               - -
  <FREE>         -:-   708K -                                                 - -
da1              0:188  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:189  14G freebsd-zfs                     gpt/cache-transcend <ZFS>
da2              0:202 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          0:207 466G freebsd-zfs                           gpt/Transcend <ZFS>
da3              0:210  29G GPT                                               - -
  da3p1          0:211  29G freebsd-zfs                        gpt/cache-august <ZFS>
% geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4
   descr: HGST HTS721010A9E630
   lunid: 5000cca8c8f669d2
   ident: JR1000D33VPSBE
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: hp DVDRAM GT80N
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 15518924800 (14G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   lunname: KingstonDataTraveler 3.0
   lunid: KingstonDataTraveler 3.0
   ident: 60A44C3FACC93110B9970045
   rotationrate: 0
   fwsectors: 63
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: StoreJet Transcend
   lunid: 5000000000000001
   ident: X3E1SAKRS
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da3
Providers:
1. Name: da3
   Mediasize: 30943995904 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   ident: E0D55EA1C84FF390A9500FDA
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

% zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Sep 12 16:29:32 2021
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      0
        Free on Write:                          0
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         43.15   GiB
        Decompressed Data Size:                 92.29   GiB
        Compression Factor:                     2.14
        Header Size:                    0.17%   162.51  MiB

L2 ARC Breakdown:                               95.91   k
        Hit Ratio:                      65.27%  62.60   k
        Miss Ratio:                     34.73%  33.31   k
        Feeds:                                  1.27    k

L2 ARC Writes:
        Writes Sent:                    100.00% 361

------------------------------------------------------------------------

% uptime
 4:29p.m.  up 23 mins, 5 users, load averages: 0.13, 0.29, 0.28
%
  • hit ratio 76.39% after around seventeen minutes' uptime
  • fell to 65.27% after a 2.6 GiB file (GhostBSD-21.09.08.iso) on the mobile HDD at da2 was used as the input file for gdd(8) output to a flash drive that was temporarily at da0.
 
grahamperrin said:
  • hit ratio 76.39% after around seventeen minutes' uptime
  • fell to 65.27% after a 2.6 GiB file (GhostBSD-21.09.08.iso) on the mobile HDD at da2 was used as the input file for gdd(8) output to a flash drive that was temporarily at da0.
Click to expand...
On this desktop I have:

Code: 
root@Tuna2 ~# zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Sat Sep 18 18:01:26 2021
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      39.34   k
        Free on Write:                          284
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         60.24   GiB
        Decompressed Data Size:                 136.00  GiB
        Compression Factor:                     2.26
        Header Size:                    0.36%   504.84  MiB

L2 ARC Evicts:
        Lock Retries:                           2
        Upon Reading:                           0

L2 ARC Breakdown:                               5.02    m
        Hit Ratio:                      97.98%  4.92    m
        Miss Ratio:                     2.02%   101.62  k
        Feeds:                                  135.40  k

L2 ARC Writes:
        Writes Sent:                    100.00% 11.09   k

------------------------------------------------------------------------

Up 2 days, but this is persistent L2ARC over boots...
 
From a link above (Single Drive ZFS) I read that ZFS is not recommended for single drive mostly because there is no repairing tools and if file structure is broken, all data will be lost. The question is - is there good repairing tool for UFS? Something like GetDataBack for NTFS.
 
6502 said:
… repairing tool for UFS? …
Click to expand...

Something other than fsck_ffs(8)?

(Was UFS a typo?)

… a link above (Single Drive ZFS)
Click to expand...

<https://www.truenas.com/community/threads/single-drive-zfs.35515/> (2015-07)

I read that ZFS is not recommended for single drive mostly because …
Click to expand...

Single-device pools can be great. Amongst my own experiences, with added emphasis:

grahamperrin said:
… cat finds it simpler to occasionally tread on the massive power button of the dock that's conveniently locked-in behind the laptop display, where he likes to sit and watch me.

zpool status consequently reports catastrophic failure of the pool that's on a mobile hard disk drive connected via USB. Numerous reported catastrophes but never a truly permanent error. Thank you, OpenZFS. It's a miracle. …
Click to expand...

Things can be greater with multiple devices per pool.

Bookmarked, although I have not yet listened:

 
ZFS on a single drive doesn't give you all the fancy redundancy features you get with multiple drives (you can have some protection from partial failures via the copies parameter). There's still a whole host of reasons to use it on a single drive. Checksums over everything, compression, flexible snapshots and easy backup, boot environments, multiple file systems with different policies set, hard UTF-8 enforcement, and so on.

It's not any "less safe" than UFS in this scenario.
 
chungy said:
ZFS on a single drive doesn't give you all the fancy redundancy features you get with multiple drives (you can have some protection from partial failures via the copies parameter). There's still a whole host of reasons to use it on a single drive. Checksums over everything, compression, flexible snapshots and easy backup, boot environments, multiple file systems with different policies set, hard UTF-8 enforcement, and so on.

It's not any "less safe" than UFS in this scenario.
Click to expand...
Agree! Also snapshots, temporarily mirroring for a replica, zfs send and other reasons.

Personally I am running ZFS on this very notebook I am writing this message. Also on Bhyve VM-s where I have ZFS just on a single file...
 
Yup, I have moved from UFS to ZFS on my last "single drive" workstation recently. It was painful to wait for fsck on 4TB space full of small junk.. but UFS on hardware RAID-0 was noticably faster than ZFS STRIPE(0). But with ZFS I neved had to fsck, can divide my pool into different mountpoints with compression make cyclic snapshots often of what is importand and less often for what is less important etc etc.

However I noticed on a test drive with possible badblocks that ZFS just crashed detached drive and said it will not work with it.. although badblocks utility did not report any problems in that area.
 
cederom said:
However I noticed on a test drive with possible badblocks that ZFS just crashed detached drive and said it will not work with it.. although badblocks utility did not report any problems in that area.
Click to expand...
Even if you try to import it read only and no mount?
 
Argentum said:
Even if you try to import it read only and no mount?
Click to expand...
That was just a stress test drive to see how ZFS manages weared disks.. to see what I could expect on single disk install with no RAID backup sectors.. did not happen on a production hardware and I hope it never happens.. but I will remember the read-only recovery access thank you :)

The disk was taken out of 3TB ~5 year old NAS working 24/7/365.. it had bad sectors around end of disk last sectors confirmed with badbkocks so GPT was not really possible to use it anymore.. but I did 2.5TB partition and put ZFS over there.. dumping some other hard disk image with dd around 500GB ZFS told that it will not work with that disk anymore. Did not really try to recover data as I did not loose anything it was just an exercise.. and I am pretty happy to know that disk is broken rather sooner than later :) I can share that drive if anyone can make use of it for testing too :)
 
grahamperrin said:
OpenZFS: All about the cache vdev or L2ARC | Klara Inc. (undated)
Click to expand...
Minor note: I saw that "undated" as well, as a date might indicate how usefull/current it is (the source is very reputable).
This article is dated: june 26 2020.

The date is only shown in the (long) article list overview*: Recent articles on FreeBSD
You'll have to extend the view though to get to the overview of the article.

___
* I prefer the date at the article itself—or at both places.
 
You must log in or register to reply here.
Back
Top