Solved ZFS: panic after trim; zroot no longer imports, stuck in boot/panic loop

[rowan194]

"SOLVED" by importing the pool with -o readonly=on and copying data off. Trying to import read/write consistently results in a panic that seems to be related to an ongoing trim process.

Now have to reinstall.

===

I did zpool trim zroot. Status showed that it was trimming, but the machine panicked about 10 seconds later. It's now stuck in a loop, where it panics as it tries to mount zroot.

PANIC: VERIFY0(metaslab_load(msp)) failed (0 == 97)

Booting rescue then attempting to import also results in a panic. I did freebsd-update on rescue just in case there was a bugfix; no change.

In the stack trace, the last call before spl_panic is vdev_trim_calculate_progress, so I presume there's an ongoing trim which is causing the panic.

Off the top of my head, some options I can think of are:

1. Some magical way to cancel the trim process, without needing to mount the pool first
2. Read the data from the pool (via some combination of mount options, or some other recovery tool) so I can reinstall then copy. I do have a backup, but it's not that current; last resort only.
3. Try disconnecting each side of the mirror, in case one drive has completed the trim, and is able to mount

I'll try #3 now, but #1 and #2 (and any other suggestions) are what I need help with.

Any ideas? Thanks.

 

[ralphbsz]

Is the data valuable? If yes, make sure your attempts at recovery do not make things worse. Every time you do an attempt at zfs import, you are running the risk of further modifying already corrupted data. I would leave the system alone, until you have found some expertise on how (and whether) to recover it.

Old joke (which is true): How do you administer a computer? You hire a man and a dog. The man is there to feed the dog. The dog is there to bite the man if the tries to touch the computer.

If the data is not terribly valuable: Think about whether simply reinstalling and restoring from backup is a better idea. You can start that task using new disks while figuring out whether data can be recovered from the damaged system.

Final question: What version of FreeBSD and ZFS were you running? Look in the bug databases whether there are known issues that could cause the symptoms you are seeing.

 

[rowan194]

Old joke (which is true): How do you administer a computer? You hire a man and a dog. The man is there to feed the dog. The dog is there to bite the man if the tries to touch the computer.

That's a good one! Never heard that.

I managed to mount the pool using readonly=on, so I have now recovered the data (it's more configs, packages, custom scripts etc), and I'm preparing to install 14.3R from scratch.

FYI, the installed version of FreeBSD was 14.1R or 14.2R, and the rescue is 14.3R. I wonder if there's a single "trim in progress" flag somewhere that would have likely solved (albeit temporarily?) this issue.