ZFS Missing pool | failed to unpack label

lucaszm · Mar 30, 2022

Hello,

After a server crash I've been unable to access a ZFS pool from my server (which is running TrueNAS, formerly known as FreeNAS).
The pool is a single disk, there is no mirror/stripe.
I'm currently trying to import on a different machine, but the pool isn't appearing anywhere and doesn't import.

Here are a few commands that I've tried:

- zpool status

Code:

no pools available

- zpool import

Code:

no pools available to import

- zpool import Media (name of my pool)

Code:

cannot import 'Media': no such pool available

- zpool import -d /dev/sdc2

Code:

no pools available to import

I've also tried a couple of other "zpool import", using different flags (-a, -f, etc), but none could find the pool.

Disk appears normally, there are no errors according to S.M.AR.T. and the partitions are still there:

(Partition 1 is something specific to TrueNAS/FreeNAS).

I've tried a couple of zdb commands, to check if the labels were there, and that seems to be the problem, as every command that I tried said "failed to unpack label".
- zdb -l /dev/sdc and zdb -l /dev/sdc2

Code:

failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

If I hexdump the partition I can see that there are still some pool information there.

- dd if=/dev/sdc2 | hd | more

Code:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00003fd0  00 00 00 00 00 00 00 00  11 7a 0c b1 7a da 10 02  |.........z..z...|
00003fe0  3f 2a 6e 7f 80 8f f4 97  fc ce aa 58 16 9f 90 af  |?*n........X....|
00003ff0  8b b4 6d ff 57 ea d1 cb  ab 5f 46 0d db 92 c6 6e  |..m.W...._F....n|
00004000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004400  01 01 00 00 00 00 00 00  00 00 00 01 00 00 00 24  |...............$|
00004410  00 00 00 20 00 00 00 07  76 65 72 73 69 6f 6e 00  |... ....version.|
00004420  00 00 00 08 00 00 00 01  00 00 00 00 00 00 13 88  |................|
00004430  00 00 00 24 00 00 00 20  00 00 00 04 6e 61 6d 65  |...$... ....name|
00004440  00 00 00 09 00 00 00 01  00 00 00 05 4d 65 64 69  |............Medi|
00004450  61 00 00 00 00 00 00 24  00 00 00 20 00 00 00 05  |a......$... ....|
00004460  73 74 61 74 65 00 00 00  00 00 00 08 00 00 00 01  |state...........|
00004470  00 00 00 00 00 00 00 00  00 00 00 20 00 00 00 20  |........... ... |
00004480  00 00 00 03 74 78 67 00  00 00 00 08 00 00 00 01  |....txg.........|
00004490  00 00 00 00 00 98 b7 6f  00 00 00 28 00 00 00 28  |.......o...(...(|
000044a0  00 00 00 09 70 6f 6f 6c  5f 67 75 69 64 00 00 00  |....pool_guid...|
000044b0  00 00 00 08 00 00 00 01  6e 0a 48 e6 0b fb 8b 16  |........n.H.....|
000044c0  00 00 00 24 00 00 00 20  00 00 00 06 65 72 72 61  |...$... ....erra|
000044d0  74 61 00 00 00 00 00 08  00 00 00 01 00 00 00 00  |ta..............|
000044e0  00 00 00 00 00 00 00 24  00 00 00 20 00 00 00 06  |.......$... ....|
000044f0  68 6f 73 74 69 64 00 00  00 00 00 08 00 00 00 01  |hostid..........|
00004500  00 00 00 00 c3 d3 d8 57  00 00 00 20 00 00 00 28  |.......W... ...(|
00004510  00 00 00 08 68 6f 73 74  6e 61 6d 65 00 00 00 09  |....hostname....|
00004520  00 00 00 01 00 00 00 00  00 00 00 24 00 00 00 28  |...........$...(|
00004530  00 00 00 08 74 6f 70 5f  67 75 69 64 00 00 00 08  |....top_guid....|
00004540  00 00 00 01 8f 54 f5 d8  95 4f cb d1 00 00 00 20  |.....T...O..... |
00004550  00 00 00 20 00 00 00 04  67 75 69 64 00 00 00 08  |... ....guid....|
00004560  00 00 00 01 8f 54 f5 d8  95 4f cb d1 00 00 00 2c  |.....T...O.....,|
00004570  00 00 00 28 00 00 00 0d  76 64 65 76 5f 63 68 69  |...(....vdev_chi|
00004580  6c 64 72 65 6e 00 00 00  00 00 00 08 00 00 00 01  |ldren...........|
00004590  00 00 00 00 00 00 00 01  00 00 02 48 00 00 00 38  |...........H...8|
000045a0  00 00 00 09 76 64 65 76  5f 74 72 65 65 00 00 00  |....vdev_tree...|
000045b0  00 00 00 13 00 00 00 01  00 00 00 00 00 00 00 01  |................|
000045c0  00 00 00 20 00 00 00 20  00 00 00 04 74 79 70 65  |... ... ....type|
000045d0  00 00 00 09 00 00 00 01  00 00 00 04 64 69 73 6b  |............disk|
000045e0  00 00 00 20 00 00 00 20  00 00 00 02 69 64 00 00  |... ... ....id..|
000045f0  00 00 00 08 00 00 00 01  00 00 00 00 00 00 00 00  |................|
00004600  00 00 00 20 00 00 00 20  00 00 00 04 67 75 69 64  |... ... ....guid|
00004610  00 00 00 08 00 00 00 01  8f 54 f5 d8 95 4f cb d1  |.........T...O..|
00004620  00 00 00 4c 00 00 00 48  00 00 00 04 70 61 74 68  |...L...H....path|
00004630  00 00 00 09 00 00 00 01  00 00 00 2f 2f 64 65 76  |...........//dev|
00004640  2f 67 70 74 69 64 2f 32  65 61 66 33 30 36 37 2d  |/gptid/2eaf3067-|
00004650  63 33 30 38 2d 31 31 65  61 2d 61 30 38 36 2d 63  |c308-11ea-a086-c|
00004660  38 36 30 30 30 62 66 38  32 38 34 00 00 00 00 60  |86000bf8284....`|
00004670  00 00 00 60 00 00 00 09  70 68 79 73 5f 70 61 74  |...`....phys_pat|
00004680  68 00 00 00 00 00 00 09  00 00 00 01 00 00 00 3a  |h..............:|
00004690  69 64 31 2c 65 6e 63 40  6e 33 30 36 31 36 38 36  |id1,enc@n3061686|
000046a0  33 36 39 36 35 36 64 33  30 2f 74 79 70 65 40 30  |369656d30/type@0|
000046b0  2f 73 6c 6f 74 40 33 2f  65 6c 6d 64 65 73 63 40  |/slot@3/elmdesc@|
000046c0  53 6c 6f 74 5f 30 32 2f  70 32 00 00 00 00 00 2c  |Slot_02/p2.....,|
000046d0  00 00 00 28 00 00 00 0e  6d 65 74 61 73 6c 61 62  |...(....metaslab|
000046e0  5f 61 72 72 61 79 00 00  00 00 00 08 00 00 00 01  |_array..........|
000046f0  00 00 00 00 00 00 00 26  00 00 00 2c 00 00 00 28  |.......&...,...(|
00004700  00 00 00 0e 6d 65 74 61  73 6c 61 62 5f 73 68 69  |....metaslab_shi|
00004710  66 74 00 00 00 00 00 08  00 00 00 01 00 00 00 00  |ft..............|
00004720  00 00 00 23 00 00 00 24  00 00 00 20 00 00 00 06  |...#...$... ....|
00004730  61 73 68 69 66 74 00 00  00 00 00 08 00 00 00 01  |ashift..........|
00004740  00 00 00 00 00 00 00 0c  00 00 00 24 00 00 00 20  |...........$... |
00004750  00 00 00 05 61 73 69 7a  65 00 00 00 00 00 00 08  |....asize.......|
00004760  00 00 00 01 00 00 03 a3  01 34 00 00 00 00 00 24  |.........4.....$|
00004770  00 00 00 20 00 00 00 06  69 73 5f 6c 6f 67 00 00  |... ....is_log..|
00004780  00 00 00 08 00 00 00 01  00 00 00 00 00 00 00 00  |................|
00004790  00 00 00 20 00 00 00 20  00 00 00 03 44 54 4c 00  |... ... ....DTL.|
000047a0  00 00 00 08 00 00 00 01  00 00 00 00 00 00 00 34  |...............4|
000047b0  00 00 00 28 00 00 00 28  00 00 00 0a 63 72 65 61  |...(...(....crea|

I ran photorec for a bit and saw that that are files on that partition, but I preferably would like to have the pool back, mostly to preserve filenames.

Can anyone help me?

Thanks in advance.

PS: English is not my native language, sorry for any mistakes.

Cath O'Deray · Apr 3, 2022

Welcome to FreeBSD Forums.

lucaszm said:
… trying to import on a different machine, …

Which version of FreeBSD, exactly?

freebsd-version -kru ; uname -aKU

Also:

zfs version

lucaszm said:
zdb -l /dev/sdc2

Code:

failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3

Can you think of any action that might have overwritten all four copies of the label?

Or, is the partition encrypted?

lucaszm · Apr 3, 2022

grahamperrin said:
Welcome to FreeBSD Forums.

Which version of FreeBSD, exactly?

freebsd-version -kru ; uname -aKU

Also:

zfs version

Can you think of any action that might have overwritten all four copies of the label?

Or, is the partition encrypted?

Hi,

I did try with TrueNAS, which has these versions:
freebsd-version -kru
12.2-RELEASE-p12

uname -aKU
FreeBSD truenas.local 12.2-RELEASE-p12 FreeBSD 12.2-RELEASE-p12 ec84e0c52a1(HEAD) TRUENAS amd64 1202000 120200

zfs version
zfs-2.0.7-1
zfs-kmod-v2022012000-zfs_e870bd8cf

Also, I tried the same commands (zdb, zpool import, etc) on mfsBSD:
freebsd-version -kru
13.0-RELEASE

uname -aKU
FreeBSD mfsbsd 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1300139 1300139

zfs version
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
zfs-2.0.0-FreeBSD_gf11b09dec
zfs-kmod-2.0.0-FreeBSD_gf11b09dec

Neither seemed to be able to import the pool.

Can you think of any action that might have overwritten all four copies of the label?

I don't remember any action that I did that could have caused that.
I did try a few zpool import and zpool export, but I never tried any zpool create, because I was afraid of overwriting any data there.

Or, is the partition encrypted?

Not that I'm aware.
But, if I'm not mistaken, there was some compression (with lz4).

Thanks in advance.

Cath O'Deray · Apr 3, 2022

lucaszm said:
… I don't remember any action that I did that could have caused that. …

OK.

Puzzling.

lucaszm said:
… After a server crash …

Was there anything extraordinary about the crash?

Can you recall what preceded the crash?

FYI, although the resolution here did involve encryption:

SOLVED - Can't find pool to import after re-install

Hi, My Boot USB died on me recently and I managed to get a fresh install on a SSD. The Boot USB was using 11.3, potentially -U4 before it died before/after the upgrade. The fresh install is using 11.3-U4. My hardware list is below: Motherboard make and model: Asus z87 mitx CPU make and...

www.truenas.com

lucaszm said:
… I ran photorec …

Should be OK. From <https://www.cgsecurity.org/wiki/PhotoRec>:

… PhotoRec uses read-only access to handle the drive or memory card you are about to recover lost data from. …

lucaszm · Apr 4, 2022

grahamperrin said:
Was there anything extraordinary about the crash?

Can you recall what preceded the crash?

Yeah, I was transfering some files to a different server over the network.
To be fair, I was already running TrueNAS under a VM, passing the HDD directly to that VM.
That probably didn't help.

grahamperrin said:
FYI, although the resolution here did involve encryption:

SOLVED - Can't find pool to import after re-install

Hi, My Boot USB died on me recently and I managed to get a fresh install on a SSD. The Boot USB was using 11.3, potentially -U4 before it died before/after the upgrade. The fresh install is using 11.3-U4. My hardware list is below: Motherboard make and model: Asus z87 mitx CPU make and...

www.truenas.com

I tried a few configuration backups that I had from TrueNAS, but none seemed to make a difference.

grahamperrin said:
Should be OK. From <https://www.cgsecurity.org/wiki/PhotoRec>:

Yeah, I was able to get some files back using PhotoRec, I was just trying to save the filenames and file structure.
It's okay, there were no critical data there.

Thanks.

lucaszm · Apr 4, 2022

Hello,

Since I didn't have much to lose, I was able to copy the bytes from the label to another position (which seemed to be the right position) usind dd.
Now I'm able to zdb -l and can see one of the labels.

root@freenas[~]# zdb -l /dev/ada1p2
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'Media'
state: 0
txg: 10008431
pool_guid: 7929230246819891990
errata: 0
hostid: 3285440599
hostname: ''
top_guid: 10328150156034165713
guid: 10328150156034165713
vdev_children: 1
vdev_tree:
type: 'disk'
id: 0
guid: 10328150156034165713
path: '/dev/gptid/2eaf3067-c308-11ea-a086-c86000bf8284'
phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'
metaslab_array: 38
metaslab_shift: 35
ashift: 12
asize: 3998634737664
is_log: 0
DTL: 52
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

However, I'm still not able to zpool import it (even with -f):

root@freenas[~]# zpool import
pool: Media
id: 7929230246819891990
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:

Media FAULTED corrupted data
gptid/2eaf3067-c308-11ea-a086-c86000bf8284 UNAVAIL corrupted data

The gptid seems to be right, though:

2. Name: ada1p2
Mediasize: 3998639460352 (3.6T)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r0w0e0
efimedia: HD(2,GPT,2eaf3067-c308-11ea-a086-c86000bf8284,0x400080,0x1d180be08)
rawuuid: 2eaf3067-c308-11ea-a086-c86000bf8284
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639460352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037127
start: 4194432

root@freenas[~]# ls -lah /dev/gptid/
total 1
dr-xr-xr-x 2 root wheel 512B Apr 4 16:18 .
dr-xr-xr-x 10 root wheel 512B Apr 4 16:18 ..
crw-r----- 1 root operator 0x72 Apr 4 16:18 2b48ce85-b709-11ea-868d-c86000bf8284
crw-r----- 1 root operator 0x88 Apr 4 16:20 2e95e42a-c308-11ea-a086-c86000bf8284
crw-r----- 1 root operator 0x76 Apr 4 16:18 2eaf3067-c308-11ea-a086-c86000bf8284

DaveQB · Feb 18, 2023

I have this same problem; some old disks that are single ZFS that FreeBSD (13.1) can't import and can't read (with zdb, failed to unpack error). It can see the partitions with fdisk, but not in the /dev file system:

Code:

$ ls /dev/da2*
/dev/da2

$ fdisk /dev/da2
******* Working on device /dev/da2 *******
parameters extracted from in-core disklabel are:
cylinders=243201 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=243201 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 3907029167 (1907729 Meg), flag 0
        beg: cyl 1023/ head 255/ sector 63;
        end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

My solution was to plug them into a Linux computer which had no issues reading the disks with zdb and able to import (with the -d switch). Then export the disks and try again on the FreeBSD computer which could them import them. Weird.

FreeBSD machine:
zfs-2.1.4-FreeBSD_g52bad4f23
zfs-kmod-2.1.4-FreeBSD_g52bad4f23

Linux machine:
zfs-2.1.5-1ubuntu6~22.04.1
zfs-kmod-2.1.4-0ubuntu0.1