I unfortunately dropped a piece of metal in the wrong spot on my motherboard which caused the computer to shutdown instantly.
Upon restarting the machine (TrueNAS Core 14 ) kernel would panic when it came to mounting my pool (12 disks, 2 zdev raidz2)
Booting latest FreeBSD 15 or Linux, no panic, but an I/O error when running zpool import.
Tried `zpool import -F` to rollback a bit, same error.
I tried zpool import -T to a txg I had found that matched the time at which the server turned off. After 4 days of intense disk activity, I got the same dreadful
so back on the FreeBSD 15 live disk.
when I ran a command I found on this post https://forums.freebsd.org/threads/zfs-pool-got-corrupted-kernel-panic-after-import.76485/
Right now I'm running the command `zpool import -o readonly=on -f -R /mnt -T 43683330 pool` but I don't know how long that's going to take again.
Help will be greatly appreciated.
TIA
Upon restarting the machine (TrueNAS Core 14 ) kernel would panic when it came to mounting my pool (12 disks, 2 zdev raidz2)
Booting latest FreeBSD 15 or Linux, no panic, but an I/O error when running zpool import.
Tried `zpool import -F` to rollback a bit, same error.
I tried zpool import -T to a txg I had found that matched the time at which the server turned off. After 4 days of intense disk activity, I got the same dreadful
Now, I can see all my dataset in there using zdb. So something is there.# zpool import -N -o readonly=on -f -R /mnt -F -T 43683300 pool
cannot import 'pool': one or more devices is currently unavailable
so back on the FreeBSD 15 live disk.
and to retrieve MOS configuration:root@:~ # zdb -ed pool
Dataset mos [META], ID 0, cr_txg 4, 1.26G, 1139 objects
Dataset pool/home/angela [ZPL], ID 108, cr_txg 177, 264K, 14 objects
Dataset pool/home/davenard [ZPL], ID 128890, cr_txg 14309410, 296K, 20 objects
Dataset pool/home/jyavenard [ZPL], ID 102, cr_txg 164, 272G, 417952 objects
Dataset pool/home [ZPL], ID 96, cr_txg 132, 272K, 19 objects
Dataset pool/.system/syslog-a5d713b37bdf437fb541f59b157cd837 [ZPL], ID 3716, cr_txg 28193766, 192K, 7 objects
Dataset pool/.system/cores [ZPL], ID 3462, cr_txg 28193761, 145M, 8 objects
Dataset pool/.system/webui [ZPL], ID 3588, cr_txg 28193772, 192K, 7 objects
Dataset pool/.system/samba4@update--2025-01-12-04-41--13.0-U6.2 [ZPL], ID 55293, cr_txg 37082320, 871K, 110 objects
Dataset pool/.system/samba4@update--2024-01-22-11-22--13.0-U5.3 [ZPL], ID 105724, cr_txg 30991476, 919K, 182 objects
Dataset pool/.system/samba4@update--2025-09-01-01-21--13.0-U6.7 [ZPL], ID 385, cr_txg 41060008, 887K, 98 objects
Dataset pool/.system/samba4@update--2024-07-07-23-59--13.0-U6.1 [ZPL], ID 66894, cr_txg 33864198, 887K, 151 objects
Dataset pool/.system/samba4@update--2025-03-16-11-12--13.0-U6.4 [ZPL], ID 10681, cr_txg 38168608, 935K, 104 objects
Dataset pool/.system/samba4 [ZPL], ID 392, cr_txg 28193764, 983K, 94 objects
Dataset pool/.system/services [ZPL], ID 3205, cr_txg 28193774, 192K, 7 objects
Dataset pool/.system/configs-a5d713b37bdf437fb541f59b157cd837 [ZPL], ID 2822, cr_txg 28193770, 315M, 2511 objects
Dataset pool/.system/rrd-a5d713b37bdf437fb541f59b157cd837 [ZPL], ID 2437, cr_txg 28193768, 132M, 2085 objects
Dataset pool/.system [ZPL], ID 657, cr_txg 28193759, 14.9M, 53 objects
Dataset pool/data/web [ZPL], ID 8386, cr_txg 27924358, 253G, 130139 objects
Dataset pool/data/music [ZPL], ID 126, cr_txg 393, 176K, 7 objects
Dataset pool/data/photos [ZPL], ID 162, cr_txg 456, 176K, 7 objects
Dataset pool/data/images [ZPL], ID 120, cr_txg 384, 390M, 1560 objects
Dataset pool/data/videos/movies [ZPL], ID 144, cr_txg 422, 103G, 537 objects
Dataset pool/data/videos/trailers [ZPL], ID 156, cr_txg 445, 176K, 7 objects
Dataset pool/data/videos/TV [ZPL], ID 138, cr_txg 413, 85.5G, 437 objects
Dataset pool/data/videos/recordings [ZPL], ID 150, cr_txg 433, 6.66T, 10635 objects
Dataset pool/data/videos [ZPL], ID 132, cr_txg 404, 264K, 16 objects
Dataset pool/data [ZPL], ID 114, cr_txg 376, 256K, 12 objects
Dataset pool/downloads [ZPL], ID 180, cr_txg 520, 91.5G, 3246 objects
Dataset pool/backup/www.avenard.org [ZPL], ID 629, cr_txg 2558121, 262G, 496406 objects
Dataset pool/backup/DominiquesiPro [ZPL], ID 128837, cr_txg 14309316, 192K, 7 objects
Dataset pool/backup/jya7980xe [ZPL], ID 697, cr_txg 5820092, 2.97T, 73579 objects
Dataset pool/backup/macbookair13/backup [ZPL], ID 1208, cr_txg 24973734, 125G, 3963 objects
Dataset pool/backup/macbookair13/jyavenard [ZPL], ID 86854, cr_txg 43590857, 192K, 7 objects
Dataset pool/backup/macbookair13 [ZPL], ID 790, cr_txg 4369423, 328K, 17 objects
Dataset pool/backup/hass [ZPL], ID 847, cr_txg 8743534, 189G, 5100 objects
Dataset pool/backup/lenovo13 [ZPL], ID 90349, cr_txg 20217124, 10.6G, 117550 objects
Dataset pool/backup/mediaserver [ZPL], ID 174, cr_txg 496, 57.5G, 1215429 objects
Dataset pool/backup/mythtv [ZPL], ID 186, cr_txg 532, 1.38G, 33 objects
Dataset pool/backup/macbookpro15 [ZPL], ID 1099, cr_txg 11690440, 881G, 80362 objects
Dataset pool/backup/mba13m2/backup [ZPL], ID 2831, cr_txg 30992041, 200K, 8 objects
Dataset pool/backup/mba13m2 [ZPL], ID 108692, cr_txg 30991189, 200K, 9 objects
Dataset pool/backup/mbp14m1/backup [ZPL], ID 108948, cr_txg 43611379, 32.3G, 37 objects
Dataset pool/backup/mbp14m1 [ZPL], ID 109455, cr_txg 43611231, 192K, 8 objects
Dataset pool/backup [ZPL], ID 168, cr_txg 488, 240K, 18 objects
Dataset pool/guest [ZPL], ID 91261, cr_txg 28039850, 979M, 9 objects
Dataset pool/vms/jira-w2jhhc_jira_clone0 [ZVOL], ID 774, cr_txg 2732864, 2.94G, 2 objects
Dataset pool/vms/hass-radar@clone_radar [ZVOL], ID 28845, cr_txg 28123669, 13.8G, 2 objects
Dataset pool/vms/hass-radar [ZVOL], ID 31234, cr_txg 28123615, 7.42G, 2 objects
Dataset pool/vms/hass-xgsk1@hass-2023-08-06_23-26 [ZVOL], ID 20705, cr_txg 28116748, 12.7G, 2 objects
Dataset pool/vms/hass-xgsk1@clone_radar [ZVOL], ID 28487, cr_txg 28123606, 13.8G, 2 objects
Dataset pool/vms/hass-xgsk1 [ZVOL], ID 52894, cr_txg 27893757, 7.01G, 2 objects
Dataset pool/vms/ubuntu-n8n5qq [ZVOL], ID 1086, cr_txg 27910227, 112K, 2 objects
Dataset pool/vms/mediaserver-evrl33@mediaserverl-2023-08-06_23-27 [ZVOL], ID 23681, cr_txg 28116752, 184G, 2 objects
Dataset pool/vms/mediaserver-evrl33 [ZVOL], ID 2323, cr_txg 28088242, 129G, 2 objects
Dataset pool/vms/jira-w2jhhc@jira_clone0 [ZVOL], ID 769, cr_txg 2732863, 2.02G, 2 objects
Dataset pool/vms/jira-w2jhhc [ZVOL], ID 766, cr_txg 2730812, 275G, 2 objects
Dataset pool/vms [ZPL], ID 728, cr_txg 2720017, 208K, 13 objects
Dataset pool/jails [ZPL], ID 584, cr_txg 318366, 184K, 10 objects
Dataset pool/iocage/download/13.2-RELEASE [ZPL], ID 8573, cr_txg 27924283, 256M, 10 objects
Dataset pool/iocage/download/11.2-RELEASE [ZPL], ID 590, cr_txg 322397, 272M, 12 objects
Dataset pool/iocage/download/11.3-RELEASE [ZPL], ID 700, cr_txg 8477876, 289M, 12 objects
Dataset pool/iocage/download/13.1-RELEASE [ZPL], ID 9278, cr_txg 27924952, 251M, 10 objects
Dataset pool/iocage/download/12.1-RELEASE [ZPL], ID 985, cr_txg 12596611, 371M, 11 objects
Dataset pool/iocage/download [ZPL], ID 541, cr_txg 258516, 192K, 12 objects
Dataset pool/iocage/releases/12.1-RELEASE/root [ZPL], ID 1028, cr_txg 12596623, 1.95G, 103912 objects
Dataset pool/iocage/releases/12.1-RELEASE [ZPL], ID 1019, cr_txg 12596622, 192K, 8 objects
Dataset pool/iocage/releases/13.1-RELEASE/root [ZPL], ID 9287, cr_txg 27924958, 892M, 17195 objects
Dataset pool/iocage/releases/13.1-RELEASE [ZPL], ID 9352, cr_txg 27924957, 192K, 8 objects
Dataset pool/iocage/releases/11.3-RELEASE/root [ZPL], ID 761, cr_txg 8477899, 1.51G, 98901 objects
Dataset pool/iocage/releases/11.3-RELEASE [ZPL], ID 755, cr_txg 8477898, 176K, 8 objects
Dataset pool/iocage/releases/11.2-RELEASE/root@transmission [ZPL], ID 605, cr_txg 322723, 1.43G, 95620 objects
Dataset pool/iocage/releases/11.2-RELEASE/root@shell [ZPL], ID 606, cr_txg 2573567, 1.43G, 95620 objects
Dataset pool/iocage/releases/11.2-RELEASE/root@sendmail [ZPL], ID 703, cr_txg 2575463, 1.43G, 95620 objects
Dataset pool/iocage/releases/11.2-RELEASE/root@couchpotato [ZPL], ID 646, cr_txg 475106, 1.43G, 95620 objects
Dataset pool/iocage/releases/11.2-RELEASE/root@teslamate [ZPL], ID 731, cr_txg 3937568, 1.43G, 95620 objects
Dataset pool/iocage/releases/11.2-RELEASE/root [ZPL], ID 602, cr_txg 322588, 1.50G, 97717 objects
Dataset pool/iocage/releases/11.2-RELEASE [ZPL], ID 596, cr_txg 322587, 176K, 8 objects
Dataset pool/iocage/releases/13.2-RELEASE/root@web [ZPL], ID 852, cr_txg 27943081, 777M, 17100 objects
Dataset pool/iocage/releases/13.2-RELEASE/root [ZPL], ID 8663, cr_txg 27924289, 777M, 17100 objects
Dataset pool/iocage/releases/13.2-RELEASE [ZPL], ID 8656, cr_txg 27924288, 192K, 8 objects
Dataset pool/iocage/releases [ZPL], ID 565, cr_txg 258524, 192K, 12 objects
Dataset pool/iocage/templates [ZPL], ID 571, cr_txg 258526, 176K, 7 objects
Dataset pool/iocage/jails/web/root@jail-2023-08-06_23-26 [ZPL], ID 20831, cr_txg 28116744, 4.97G, 329930 objects
Dataset pool/iocage/jails/web/root [ZPL], ID 924, cr_txg 27943083, 7.18G, 400180 objects
Dataset pool/iocage/jails/web@jail-2023-08-06_23-26 [ZPL], ID 20829, cr_txg 28116744, 208K, 10 objects
Dataset pool/iocage/jails/web [ZPL], ID 314, cr_txg 27943082, 224K, 10 objects
Dataset pool/iocage/jails/shell/root@ioc_update_11.3-RELEASE-p14_2023-08-04_20-15-49 [ZPL], ID 21977, cr_txg 28085309, 5.19G, 400595 objects
Dataset pool/iocage/jails/shell/root@ioc_update_11.2-RELEASE-p15_2020-07-12_13-07-09 [ZPL], ID 668, cr_txg 8494174, 2.79G, 151462 objects
Dataset pool/iocage/jails/shell/root@ioc_update_11.2-RELEASE-p9 [ZPL], ID 685, cr_txg 4560311, 2.55G, 140608 objects
Dataset pool/iocage/jails/shell/root@ioc_update_11.2-RELEASE-p15_2020-07-12_13-07-54 [ZPL], ID 674, cr_txg 8494183, 2.79G, 151469 objects
Dataset pool/iocage/jails/shell/root@ioc_update_11.3-RELEASE-p11_2023-07-26_09-08-55 [ZPL], ID 7931, cr_txg 27924398, 3.33G, 169462 objects
Dataset pool/iocage/jails/shell/root@jail-2023-08-06_23-26 [ZPL], ID 20835, cr_txg 28116744, 7.76G, 557372 objects
Dataset pool/iocage/jails/shell/root [ZPL], ID 642, cr_txg 2573569, 8.18G, 572968 objects
Dataset pool/iocage/jails/shell@ioc_update_11.3-RELEASE-p14_2023-08-04_20-15-49 [ZPL], ID 21975, cr_txg 28085309, 232K, 11 objects
Dataset pool/iocage/jails/shell@ioc_update_11.2-RELEASE-p15_2020-07-12_13-07-09 [ZPL], ID 525, cr_txg 8494174, 216K, 11 objects
Dataset pool/iocage/jails/shell@ioc_update_11.2-RELEASE-p9 [ZPL], ID 623, cr_txg 4560311, 192K, 10 objects
Dataset pool/iocage/jails/shell@ioc_update_11.3-RELEASE-p11_2023-07-26_09-08-55 [ZPL], ID 7929, cr_txg 27924398, 216K, 11 objects
Dataset pool/iocage/jails/shell@jail-2023-08-06_23-26 [ZPL], ID 20833, cr_txg 28116744, 216K, 11 objects
Dataset pool/iocage/jails/shell@ioc_update_11.2-RELEASE-p15_2020-07-12_13-07-54 [ZPL], ID 672, cr_txg 8494183, 216K, 11 objects
Dataset pool/iocage/jails/shell [ZPL], ID 635, cr_txg 2573568, 216K, 11 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p15_2022-08-18_16-10-51 [ZPL], ID 60164, cr_txg 21674773, 28.4G, 600674 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p15_2022-08-18_16-12-58 [ZPL], ID 59945, cr_txg 21674799, 28.4G, 600674 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p15_2022-08-18_16-12-36 [ZPL], ID 60050, cr_txg 21674794, 28.4G, 600674 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p15_2023-07-26_08-52-27 [ZPL], ID 8537, cr_txg 27924186, 30.8G, 601340 objects
Dataset pool/iocage/jails/sendmail/root@jail-2023-08-06_23-26 [ZPL], ID 20839, cr_txg 28116744, 30.8G, 601339 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p4_2022-08-18_16-09-07 [ZPL], ID 59519, cr_txg 21674752, 28.4G, 600379 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p4_2021-03-08_02-17-22 [ZPL], ID 824, cr_txg 12596552, 25.4G, 586619 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_11.2-RELEASE-p9_2020-07-12_13-32-40 [ZPL], ID 236, cr_txg 8494512, 2.98G, 139433 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_11.3-RELEASE-p11_2021-03-07_23-21-30 [ZPL], ID 414, cr_txg 12594445, 22.6G, 452470 objects
Dataset pool/iocage/jails/sendmail/root@ioc_update_12.2-RELEASE-p15_2022-08-18_16-13-40 [ZPL], ID 60172, cr_txg 21674808, 28.4G, 600611 objects
Dataset pool/iocage/jails/sendmail/root [ZPL], ID 714, cr_txg 2575465, 35.3G, 602246 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p15_2022-08-18_16-10-51 [ZPL], ID 60041, cr_txg 21674773, 224K, 11 objects
Dataset pool/iocage/jails/sendmail@jail-2023-08-06_23-26 [ZPL], ID 20837, cr_txg 28116744, 208K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p15_2023-07-26_08-52-27 [ZPL], ID 8535, cr_txg 27924186, 208K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p15_2022-08-18_16-12-36 [ZPL], ID 59935, cr_txg 21674794, 224K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p15_2022-08-18_16-12-58 [ZPL], ID 59943, cr_txg 21674799, 224K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_11.2-RELEASE-p9_2020-07-12_13-32-40 [ZPL], ID 193, cr_txg 8494512, 208K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p15_2022-08-18_16-13-40 [ZPL], ID 59951, cr_txg 21674808, 224K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p4_2021-03-08_02-17-22 [ZPL], ID 822, cr_txg 12596552, 224K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_11.3-RELEASE-p11_2021-03-07_23-21-30 [ZPL], ID 796, cr_txg 12594445, 208K, 11 objects
Dataset pool/iocage/jails/sendmail@ioc_update_12.2-RELEASE-p4_2022-08-18_16-09-07 [ZPL], ID 60034, cr_txg 21674752, 224K, 11 objects
Dataset pool/iocage/jails/sendmail [ZPL], ID 708, cr_txg 2575464, 208K, 11 objects
Dataset pool/iocage/jails@jail-2023-08-06_23-26 [ZPL], ID 20827, cr_txg 28116744, 192K, 10 objects
Dataset pool/iocage/jails [ZPL], ID 553, cr_txg 258520, 192K, 10 objects
Dataset pool/iocage/log [ZPL], ID 559, cr_txg 258522, 384K, 11 objects
Dataset pool/iocage/images [ZPL], ID 547, cr_txg 258518, 176K, 7 objects
Dataset pool/iocage [ZPL], ID 535, cr_txg 258514, 10.6M, 483 objects
Dataset pool [ZPL], ID 21, cr_txg 1, 240K, 15 objects
MOS object 753 (DSL dir clones) leaked
Verified large_blocks feature refcount of 0 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 100 is correct
Verified encryption feature refcount of 0 is correct
Verified project_quota feature refcount of 100 is correct
Verified redaction_bookmarks feature refcount of 0 is correct
Verified redacted_datasets feature refcount of 0 is correct
Verified bookmark_written feature refcount of 0 is correct
Verified livelist feature refcount of 0 is correct
Verified zstd_compress feature refcount of 0 is correct
Does this last one indicate the last txg is 43203194 ?root@:~ # zdb -eC pool
MOS Configuration:
version: 5000
name: 'pool'
state: 0
txg: 43203194
pool_guid: 9742808535407341325
errata: 0
hostid: 623965209
hostname: 'supernas.local'
com.delphix:has_per_vdev_zaps
vdev_children: 2
vdev_tree:
type: 'root'
id: 0
guid: 9742808535407341325
create_txg: 4
children[0]:
when I ran a command I found on this post https://forums.freebsd.org/threads/zfs-pool-got-corrupted-kernel-panic-after-import.76485/
I'd like to retry mounting the pool with a rollback as described https://www.perforce.com/blog/pdx/openzfs-pool-import-recovery but they don't indicate how they determinate the latest "good" txg (in their example it was 50).zdb -ul /dev/da0 > /tmp/uberblocks.txt`
gave me much later txg :
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
------------------------------------
LABEL 2 (Bad label cksum)
------------------------------------
Uberblock[64]
magic = 0000000000bab10c
version = 5000
txg = 43683328
guid_sum = 9645404203117630058
timestamp = 1769858073 UTC = Sat Jan 31 11:14:33 2026
bp = DVA[0]=<0:148d2c4f5000:3000> DVA[1]=<1:113301fac000:3000> DVA[2]=<0:ffba8ced000:3000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=800L/800P birth=43683328L/43683328P fill=1138 cksum=00000002a033deb1:000004e0ab7140fd:00048aa393ecc229:02d37c89285a7f87
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 2 3
Right now I'm running the command `zpool import -o readonly=on -f -R /mnt -T 43683330 pool` but I don't know how long that's going to take again.
Help will be greatly appreciated.
TIA