ZFS on 14.0 boot panic since new year

W

wb.kloke

Since 1.1.24 my ZFS system panics in the boot process even when booted single-user.
The system was orderly shut down before.

Some strange observations:

On linux, no problems accessing the zroot pool,
On a NFS booted FreeBSD (13.2), the pool is accessible readonly,
When I try importing the pool without readonly=on, the system panics immediately.

One time, after using some commands from a forum thread in Jan 2023, I got access again,
but this is not repeatable any more. Then I renamed the /boot/zfs/zpool.cache file. This did not help either:

Now I am going to ditch ZFS use for FreeBSD boot, if there is no way to recover other than use the backup.
 
Welcome to The FreeBSD Forums.

Did 14.0-RELEASE panic before 1st January?

Can you share a photograph of the panic?

Can you tell us more about the hardware?

Were packages upgraded recently?
 
To Graham questions:

> Did 14.0-RELEASE panic before 1st January?

No. Or rather perhaps, On some days I had to boot again, but seldom.

> Can you share a photograph of the panic?

Sorry, no. The screen just became black, and a reboot started.

> Can you tell us more about the hardware?
dmesg from the NFS started system (abbreviated)

Code: 
FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(vga): resolution 640x480
CPU: AMD PRO A10-9700 R7, 10 COMPUTE CORES 4C+6G     (3493.55-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x660f51  Family=0x15  Model=0x65  Stepping=1
...
real memory  = 34359738368 (32768 MB)
avail memory = 32753799168 (31236 MB)
....
nvme0: Allocated 61MB host memory buffer
nvd0: <KBG40ZNV256G KIOXIA> NVMe namespace
nvd0: 244198MB (500118192 512 byte sectors)
...
NFS ROOT: 192.168.44.16:/srv/FreeBSD
re0: link state changed to UP
[drm] amdgpu kernel modesetting enabled.
drmn0: <drmn> on vgapci0
vgapci0: child drmn0 requested pci_enable_io
vgapci0: child drmn0 requested pci_enable_io
sysctl_warn_reuse: can't re-use a leaf (hw.dri.debug)!
[drm] initializing kernel modesetting (CARRIZO 0x1002:0x9874 0x103C:0x8463 0xE2).
...

The zfs pool resides on the nvme disk, in 14.0, the driver is nda, of course.
The only problem I encountered in upgrading to 14.0 was that I needed to use amdgpu 5.10 instead of 5.15

The system was in use daily without problems. I don't know whether the year change has anything to do with the crash, but it happened on the first boot on Jan 1st,after orderly shutdown at 1:30 same night.

> pkgs upgraded?

Yes. If I remember correctly, I I did a 'pkg upgrade', which insalled a new chromium browser.
I don't think that this affected the system, as I got the system up for one time, probably very accidentally.[/ICODE]
 
loader.conf is empty on the target system.

BTW the crash happens (booting multi-user) after a line telling about loading the kernel modules.
In single user very late, shortly before opening the shell.

I am just in the process of zfs-send of the whole pool. No problems are reported so far.
But importing the pool without setting readonly=on reliably crashes any FreeBSD system, but not Linux.
 
Did you upgrade you system (from 13.x for example) and forgot to update your kernel modules (from ports or packages)?
 
> Did you upgrade you system (from 13.x for example) and forgot to update your kernel modules (from ports or packages)?

The 14.0 system was working fine for some days. As said earlier, the only quirk was the amdgpu module.

But, for the state of the zroot pool, it now crashes immediately both a 13.2 (via nfs booted) system when imported without readonly=on, as the 14.0 from itself.

As the data in the pool seem to be consistent (by scrub), i am quite astonished, that there is no way documented way to make the pool writable again.
 
Let's make the dates clear:

From 'zfs list' on Linux:

Code: 
zroot/ROOT/13.2-RELEASE_2023-11-29_180827        1M   188G     18.5G  /mnt
zroot/ROOT/14.0-RELEASE-p1_2023-12-08_154936     8K   188G     21.6G  /mnt
zroot/ROOT/14.0-RELEASE_2023-11-29_181054        8K   188G     18.6G  /mnt
zroot/ROOT/14.0-RELEASE_2023-11-29_182517        8K   188G     21.4G  /mnt
zroot/ROOT/14.0-RELEASE_2023-12-04_153914        8K   188G     21.3G  /mnt

These snapshots are resulting from freebsd-update. So, i upgraded to 14.0 on nov 29, the to -p1 and -p2
on 29 Nov and 4 Dec and 8 Dec. The system was used 4 weeks without problems.

The newest files in /boot/kernel are from 8 Dec. Same for /bin /sbin /usr/bin /usr/sbin, /ib, /usr/lib.
In /usr/local/bin 21 Dec, in /usr/local/sbin and /usr/local/lib 19Dec.
 
Crash info will be key to begin understanding the problem(s).

Do you have an installation of FreeBSD with crash reporting enabled?
 
grahamperrin said:
Crash info will be key to begin understanding the problem(s).

Do you have an installation of FreeBSD with crash reporting enabled?
Click to expand...
Sorry, no. On one time, I got a message about segment missing, but I had no camera ready to take a screen shot.

Now I have closed the case, as I needed the system and I have no other M2 card, so I destroyed the pool and restored it from a backup copy.

I learned a couple of things about zfs and uefi booting, and hope the disaster will not hit me again soon.
 
You must log in or register to reply here.
Back
Top