ZFS Regular panics in zfs' zio_buf_alloc and vm_fault_lookup

E

enumeration

Hi *,

my system is regularly (approx. once daily) panicing with one of the following stacktraces:

Code: 
panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe01a7041000
cpuid = 3
time = 1788555071
KDB: stack backtrace:
#0 0xffffffff80bb434d at kdb_backtrace+0x5d
#1 0xffffffff80b65bb1 at vpanic+0x161
#2 0xffffffff80b65a43 at panic+0x43
#3 0xffffffff80ee96b6 at vm_fault+0x1696
#4 0xffffffff80ee7f31 at vm_fault_trap+0x81
#5 0xffffffff8105abda at trap_pfault+0x20a
#6 0xffffffff810315b8 at calltrap+0x8
#7 0xffffffff82203a52 at traverse_visitbp+0x882
#8 0xffffffff822037de at traverse_visitbp+0x60e
#9 0xffffffff822037de at traverse_visitbp+0x60e
#10 0xffffffff822037de at traverse_visitbp+0x60e
#11 0xffffffff822037de at traverse_visitbp+0x60e
#12 0xffffffff822037de at traverse_visitbp+0x60e
#13 0xffffffff82204362 at traverse_dnode+0xd2
#14 0xffffffff82203620 at traverse_visitbp+0x450
#15 0xffffffff82202d5e at traverse_impl+0x32e
#16 0xffffffff82202efb at traverse_dataset_destroyed+0x2b
#17 0xffffffff821e3d0d at bptree_iterate+0x19d
Uptime: 5h47m37s
or
Code: 
panic: VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (36028797018963967 < 32768)

cpuid = 2
time = 1788555840
KDB: stack backtrace:
#0 0xffffffff80bb434d at kdb_backtrace+0x5d
#1 0xffffffff80b65bb1 at vpanic+0x161
#2 0xffffffff821adc4a at spl_panic+0x3a
#3 0xffffffff8237126b at zio_buf_alloc+0x5b
#4 0xffffffff821e74b0 at abd_alloc_linear+0x70
#5 0xffffffff821edf74 at arc_hdr_alloc_abd+0x84
#6 0xffffffff821ec815 at arc_read+0x645
#7 0xffffffff8222e18d at traverse_prefetch_metadata+0x13d
#8 0xffffffff8222da52 at traverse_visitbp+0x882
#9 0xffffffff8222d7de at traverse_visitbp+0x60e
#10 0xffffffff8222d7de at traverse_visitbp+0x60e
#11 0xffffffff8222d7de at traverse_visitbp+0x60e
#12 0xffffffff8222d7de at traverse_visitbp+0x60e
#13 0xffffffff8222d7de at traverse_visitbp+0x60e
#14 0xffffffff8222e362 at traverse_dnode+0xd2
#15 0xffffffff8222d620 at traverse_visitbp+0x450
#16 0xffffffff8222cd5e at traverse_impl+0x32e
#17 0xffffffff8222cefb at traverse_dataset_destroyed+0x2b
Uptime: 11m33s
Dumping 1425 out of 32699 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
More context:
  • It only happens when pool "aoki" is mounted, which is my data pool, not the root pool.
  • I have no idea when it started to happen. Back then I had entirely different suspicions about why the system rebooted every night, and most of the time I did not have the capacities to investigate.
  • One popular culprit would be the RAM. However, Memtest never identified any problems.
About the system:
Code: 
FreeBSD wren 14.4-RELEASE-p3 FreeBSD 14.4-RELEASE-p3 GENERIC amd64
The problem was there with prior releases as well.

The problem got more intense over time. Zpool scrub cannot finish any more (system panics before scrub is done). There are permanent errors listed.
Code: 
pool: aoki
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: [URL]https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A[/URL]
  scan: scrub in progress since Tue Sep  8 19:25:48 2026
        1.09T / 8.82T scanned at 12.1G/s, 0B / 8.82T issued
        0B repaired, 0.00% done, no estimated completion time
config:

        NAME                    STATE     READ WRITE CKSUM
        aoki                    ONLINE       0     0     0
          raidz1-0              ONLINE       0     0     0
            label/VGKBD7NK.eli  ONLINE       0     0     0
            label/VYG0BMJR.eli  ONLINE       0     0     0
            label/WSD4XENT.eli  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        aoki/backup/canterlot/home/spike:<0x1b900>
        <0xffffffffffffffff>:<0x1b900>

I don't really know what to do with this, and how to continue debugging.
Does anyone have any ideas or good hints?

I considered abandoning the pool and building an entirely fresh one with new hard disks, then copying file-by-file. But the current hardware prices ...
 
You have a corrupted block pointer somewhere in the pool (maybe more than one).
It is reached while ZFS is doing background freeing of blocks belonging to a destroyed snapshot or filesystem or zvol.

The most straightforward solution, although resource intensive, is to import the pool in read-only mode and replicate it via zfs send + recv to a new pool.
 
Thanks a lot, that makes sense!

Are there any other options other than replication that I can try? (That would be quite costly these days 😬)
 
Hi,

I tried this. Of course, now zfs send complains about an Input/Output error

sh: 
# zfs send aoki/landfill/videos@2023-03-16 | pv | zfs recv gideon/media/videos                                                                                                                                         
warning: cannot send 'aoki/landfill/videos@2023-03-16': Input/output error
34.8MiB 0:00:00 [72.6MiB/s] [    <=>                                                                                                                                                                                                                                         ]
cannot receive new filesystem stream: incomplete stream
 
Hi enumeration,

The VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) panic inside zio_buf_alloc / arc_read almost always points to a corrupt block pointer (DVA/offset) in ZFS metadata on disk where ZFS tries to allocate an impossibly large buffer size based on corrupted record size headers. Since memtest passed clean, the issue is on-disk metadata structure rather than faulty system RAM.

Since zpool scrub panics the box when it hits the corrupt block, your top priority should be salvaging whatever data isn't backed up:

1. Try importing the pool strictly read-only and unmounted without running transactions:
zpool import -o readonly=on -N -f aoki
Then mount individual datasets one by one (zfs mount aoki/dataset) and rsync your critical data to another drive. Read-only import prevents ZFS from attempting write-side ARC allocations and background async destroy tasks that trigger the panic.
2. You can also test a non-destructive rewind check:
zpool import -f -F -n aoki
(the -n simulates rewind to see if an earlier Uberblock contains clean tree state without actually committing the rollback).

What does zpool status -v aoki show for permanent errors? Does it list specific file paths or unresolvable metadata object IDs?
 
Hey, thank you two!

About the permanent errors: as included in the original post, `zpool status -v` lists object IDs:

Code: 
aoki/backup/canterlot/home/spike:<0x1b900>
<0xffffffffffffffff>:<0x1b900>

However, since I started trying `zfs send`, it also started listing some specific file paths (and then aborted the sending with "Input/output error"). All of these files are included in my backups, so I'm lucky about that.

So, about what I've been doing in the meantime: My backup capacities are not large enough for everything, they were always just provisioned for the most-important things. And I can't get more disks right now, but I did find some extra disk space on other machines. Still not enough for everything, so I did the risky and highly illegal thing that nobody ever should do (degrade the pool by offlining one of the raidz's vdevs, create a new pool on the freed disk, rsync files over - next step is going to be destroying the old pool and creating a new degraded raidz pool on the two remaining disks, copy everything back, destroy the intermediate pool, add the last disk and resilver). Let's just hope this works 😵‍💫

Since I'm already so far on this way, I didn't try `zpool import -f -F -n aoki` any more. Thank you for the suggestion though!
 
You must log in or register to reply here.
Back
Top