Solved FreeBSD sysctl vfs.zfs.dmu_offset_next_sync and openzfs/zfs issue #15526 (errata notice, FreeBSD bug 275308)

Am I to believe that this should not be an issue on FreeBSD versions lower than 14.0-RELEASE?

Reading the GitHub issue, here is what I see on one box that has been upgraded to 14.0-RELEASE (along with zpool upgrade to zfs-2.2.0-FreeBSD_g95785196f):

Code:
root@loft-bsd:~ # zpool get all zroot | grep block_cloning
zroot  feature@block_cloning          enabled                        local
root@loft-bsd:~ # sysctl vfs.zfs.dmu_offset_next_sync
vfs.zfs.dmu_offset_next_sync: 1
root@loft-bsd:~ # zpool get all zroot | grep bclone
zroot  bcloneused                     0                              -
zroot  bclonesaved                    0                              -
zroot  bcloneratio                    1.00x                          -
root@loft-bsd:~ # sysctl vfs.zfs.bclone_enabled
vfs.zfs.bclone_enabled: 0

Whereas on my server running 13.2-RELEASE-p4 with zfs-2.1.9-FreeBSD_g92e0d9d18:

Code:
root@manaha:~ # zpool get all zroot | grep block_cloning
root@manaha:~ # sysctl vfs.zfs.dmu_offset_next_sync
vfs.zfs.dmu_offset_next_sync: 1
root@manaha:~ # zpool get all zroot | grep bclone
root@manaha:~ # sysctl vfs.zfs.bclone_enabled
sysctl: unknown oid 'vfs.zfs.bclone_enabled'
 
Yes, i am able to reproduce this on my 13.2-RELEASE-p5 system.

I modified the script from github to make it FreeBSD compatible.
Code:
#!/usr/local/bin/bash
#
prefix="reproducer_${BASHPID}_"
dd if=/dev/urandom of=${prefix}0 bs=16K count=1 status=none

echo "writing files"
end=1000
h=0
for i in `seq 1 2 $end` ; do
        let "j=$i+1"
        cp ${prefix}$h ${prefix}$i
        cp ${prefix}$i ${prefix}$j
        let "h++"
done

echo "checking files"
for i in `seq 1 $end` ; do
        diff ${prefix}0 ${prefix}$i
done

When this is run with
[code]
./reproducer.sh & ./reproducer.sh & ./reproducer.sh & ./reproducer.sh &./reproducer.sh & ./reproducer.sh & ./reproducer.sh & ./reproducer.sh &./reproducer.sh & ./reproducer.sh & ./reproducer.sh & ./reproducer.sh & wait
[/code]
multiple times, sometime i get something like
Code:
...
checking files
Binary files reproducer_88319_0 and reproducer_88319_228 differ
Binary files reproducer_88337_0 and reproducer_88337_306 differ
Binary files reproducer_88319_0 and reproducer_88319_457 differ
Binary files reproducer_88319_0 and reproducer_88319_458 differ
Binary files reproducer_88337_0 and reproducer_88337_613 differ
Binary files reproducer_88337_0 and reproducer_88337_614 differ
Binary files reproducer_88319_0 and reproducer_88319_915 differ
Binary files reproducer_88319_0 and reproducer_88319_916 differ
Binary files reproducer_88319_0 and reproducer_88319_917 differ
Binary files reproducer_88319_0 and reproducer_88319_918 differ
[1]   Done                    ./reproducer.sh
...

One run takes on my machine 1m16s.

Setting
[cmd]
doas sysctl vfs.zfs.dmu_offset_next_sync=0
[/cmd]

One run just takes 4s now and the error is no more reproducible. I don't get, why this is now even faster. Just for me?
 
I was not happy to see this bug; but then who was. I just did a migration I was really wanting to avoid - jumped from 12.4 to 14.0. I followed this briefly and to my understanding it's still not known if opensolaris version is affected. I ran the script and ruled out all the false positives. Luckily enough there's only one set of private data that seems to be a valid issue - photos from 2005 of something I would not be too angry about. From that only 1 photo is of len 0 which doesn't make sense.

I suspended all my (private) backups for now.
 
I'd swear I downloaded it from 15526 github though I can't find it there. On FreeBSD I changed the bash location.

It was
Code:
#!/bin/bash

ZFS_PATH="/yourmountpoint"
RECORD_SIZE=4096

shopt -s globstar

for FILE in "$ZFS_PATH"/**; do
 if [ ! -f "$FILE" ]; then
  continue
 fi

 if !(dd if="$FILE" bs=1 count=$RECORD_SIZE 2>/dev/null | grep -q '[^[:space:]]'); then
  echo "Possible data corruption in $FILE"
 fi
done
My private backups do have 2nd location running Linux as FreeBSD is practically not able to use AX210. There I recently did 8TB backup from 12.4 to 13.2 (HW that is now running Linux) and let those data be used readonly on openzfs 2.2.0~rc3-0ubuntu4.

But maybe I spoke too soon. This script found many issues on various location of the source code, not private data only (such as python scripts of hana db servers, Linux source codes on various qemu images I used for debugging). Luckily enough this is not of any importance as I can download any of that either from corporate repo or public domain.

Those photos are the only "issue" I have right now.
 
I was not happy to see this bug; but then who was. I just did a migration I was really wanting to avoid - jumped from 12.4 to 14.0.
You jumped from pre-OpenZFS (12.4) to OpenZFS 2.2 (14.0). If you haven't upgraded your zpools at all, then I think your zpools should be still on the old version 28 (=pre Feature Flags era). If your BEs are created as per default then you should be able to revert to 12.4 AFAIK
 
Well I wrote above I did backup from 12.4 to 13.2. During backup receiving host was 13.2, pools updated to latest-greatest. That pool then was imported ro to Ubuntu server.
From what I read though it's not clear if this bug is affecting opensolaris versions too. That data, pictures from 2005, were last touched in 2009. Other data was heavily used in 2016-2018. That is all on 12.4. I never used 13.x anywhere, only at home.
 
I can't find

See Convenience links to scripts (the comment in Reddit; NB the opening post here).

This script found many issues

Not definite issues; possible issues.

At a glance: you're working with an outdated edition, maybe the first edition, of the RichardBelzer script, which is currently in the midst of 179 or more hidden items. When I worked with a more recent edition of the script, it reported 270 'Possible data corruption …' lines for my home directory. IIRC GitHub includes discussion of the script.

Instead, maybe try the bsdice script. Whether it's the best, I don't know, but (as noted in Reddit) it reported nothing from a broader scan of /usr/home.It's now scanning /.

HTH
 

Attachments

  • 1700960131503.png
    1700960131503.png
    481.8 KB · Views: 64
Via <https://old.reddit.com/r/freebsd/comments/185gohv/-/>:

Data-destroying defect found after OpenZFS 2.2.0 release • Liam Proven, The Register

Liam's article gives prominence to Ed Maste's email to the freebsd-stable list. …




Attached: a result of scanning a mobile hard disk drive, on USB, that's given to a pool where most files are VirtualBox-related.
 

Attachments

  • 2023-11-28 01.10 zfs-issue-15526-check-file t1000.txt
    22.4 KB · Views: 47
bakul thanks.


– obtained from:


– with reference to:
 

The slightly bigger picture​


Expect:
  1. merge upstream
  2. merge to FreeBSD src main (CURRENT)
  3. cherry-picks to stable and releng branches after respectable periods of time.
 
Back
Top