ZFS Need help to recover data from damaged ZFS pool

Farokh · Jan 1, 2015

Happy New Year everyone (although mine's off to a pretty poor start).

Here's my situation. I'm running a FreeNAS 9.3 (based on FreeBSD 9) system on an Intel S5000PAL based system with 16GB RAM and 3 2TB drives as a ZFS pool (the boot drive is a 16GB USB flash drive). When I restarted the machine my ZFS pool disappeared. I've tried a number of things, but to no avail. If I use zpool import, it simply exits after about 2 seconds with no message. zpool status tells me that there are no pools. gpart list shows me all 3 drives, with each drive having two partitions - one for FreeBSD swap and one for ZFS. However, glabel status only shows me 1 of the 3 drives and the 2 partitions on that drive. gpart show again shows all 3 drives and the 2 partitions for each drive.

I can't even recall the actual name of the pool since I haven't had to touch anything in many months.

Any pointers on what I can do to try and recover the pool?

Thanks!

gofer_touch · Jan 1, 2015

Hi Farokh,

Sorry to hear about the lost pool. I'm not enough a ZFS guru to really offer any advice but I do also look forward to learning from the feedback that I hope will come along as a result of your post. Were you by chance using encryption on your pool? Hopefully its a simple fix.

Farokh · Jan 1, 2015

No, no encryption, which is why I'm hoping I'll be able to recover the data.

Thanks.

doc1623 · Jan 1, 2015

I'm no expert but I guess you tried zpool import -a.

Farokh · Jan 1, 2015

Yes, I did. The problem is that the pool isn't seen by any method I've been able to find.

tingo · Jan 2, 2015

Since you didn't mention it - have you tried zpool list?

Farokh · Jan 2, 2015

zpool list tells me that there are no pools.

MeesterWood · Jan 3, 2015

Did you try zpool export tank first, and then zpool import tank?

What is the output for zpool history?

Farokh · Jan 4, 2015

One of the issues is that I don't recall the name that I gave the pool. That's one of the things I'm hoping that someone can help me figure out.

cracauer@ · Jan 5, 2015

Check the S.M.A.R.T. information on all three drives. glabel gets its default label from metadata from the disk. If the disk were to not deliver that data it would show up in gpart but glabel's automatic generation of a label would fail.

getopt · Jan 5, 2015

Farokh said:
One of the issues is that I don't recall the name that I gave the pool. That's one of the things I'm hoping that someone can help me figure out.

If you can access zpool.cache which is usually located in /boot/zfs/ do zdb –C –U /path/to/zpool.cache. You now can just read the zfs-pool name.
For further reading see zdb(8).

phoenix · Jan 6, 2015

Did you label the partitions and use the labels to create the pool? Meaning, is there anything listed in # gpart list -l for each disk? If there is, then you can try # zpool import -d /dev/gpt <poolname>

You can also check for "destroyed" pools via # zpool import -D

And, you can probably play with the following to see if the pool is listed:
# zpool import -d /dev
# zpool import -d /dev/gpt
# zpool import -d /dev/label

Farokh · Jan 7, 2015

I'll just make one post so not to clutter things up.

I have the machine apart at the moment, but last I checked the SMART status was good for all 3 of the drives.

As soon as I have time to put the machine together, I'll try the other suggestions and report back.

Thanks for the help.

Farokh · Jan 8, 2015

OK, so I put the machine together and SMART shows no errors on any of the drives.

There is no zpool.cache file. /boot/zfs/zpool.cache is a link to /data/zfs/zpool.cache, which doesn't exist.

gpart list -l gave me an error. -l is not an option for gpart(8) on my machine ( uname -a tells me:

Code:

FreeBSD freenas.local 9.3-RELEASE-p5 FreeBSD 9.3-RELEASE-p5 #1 r275790+7b75f75: Tue Dec 30 09:13:52 PST 2014     root@build3.ixsystems.com:/tank/home/jkh/build/93/FN/objs/os-base/amd64/fusion/jkh/93/FN/FreeBSD/src/sys/FREENAS.amd64  amd64).

The zpool imports phoenix suggested either do nothing, or give me errors (there's no /dev/gpt or /dev/label).

Thanks.

phoenix · Jan 8, 2015

Whoops, sorry, it's # gpart show -l

My bad.

Farokh · Jan 15, 2015

OK, here's what I got:

Code:

[root@freenas] ~# gpart show -l

=>      34  31266749  da0  GPT  (14G)
        34      1024    1  (null)  (512k)
      1058         6       - free -  (3.0k)
      1064  31265712    2  (null)  (14G)
  31266776         7       - free -  (3.5k)

=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

Thanks.

HarryE · Jan 15, 2015

Try zdb -l /dev/ada2p2. It should list the zpool name and a lot of other info.

Farokh · Jan 15, 2015

I tried zdb -l /dev/ada2p2 and got:

Code:

--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

I got the same for all the other drives.

Thanks.

HarryE · Jan 15, 2015

It seems that your disks have been repartitioned. Did you change anything in BIOS between reboots? Do you have some *.nop files in your /dev ? FreeNAS sometimes creates gnop(8) or geli(8) devices on existing drives, in such cases the partition table is displayed differently. I suggest you download and boot a FreeBSD 9.3 install image (do not install, just boot), enter the console and try to import the pool from there.

Farokh · Jan 17, 2015

No, no changes in the BIOS. I tried booting with a 9.3 CD and got the same results.

HarryE · Jan 17, 2015

Then you have to search for string pool_guid on the disk. In my case is at offset 0x000040a4 from the begining of the ZFS partition. I hope is the same for you. I use 512b/sector disk.
When you find it, determine the sector number where it resides. Substract 32(0x40a4/512) (maybe 33) from determined sector number and hopefully you have the old partition offset.
You have to manually repartition the same disk and create a freebsd-zfs partition using gpart backup /dev/ada2 > ada2.part_layout then cp ada2.part_layout ada2.my_layout.
Edit ada2.my_layout with the sector number determined above and the corresponding remaining size. Delete any other partitions, leave only one for starters. My partition look like this:

Code:

GPT 128
1 freebsd-zfs 40 5860533088

Use cat ada2.my_layout | gpart restore -F ada2
Then zdb -l /dev/ada2p1 should list your data and zpool import should list the old pool in DEGRADED state with one disk ONLINE and the other unavailable.
Repeat the steps for the other relevant disks.
HTH

gkontos · Jan 17, 2015

Farokh said:

OK, here's what I got:

Code:

[root@freenas] ~# gpart show -l

=>      34  31266749  da0  GPT  (14G)
        34      1024    1  (null)  (512k)
      1058         6       - free -  (3.0k)
      1064  31265712    2  (null)  (14G)
  31266776         7       - free -  (3.5k)

=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47k)
        128     4194304     1  (null)  (2.0G)
    4194432  3902834703     2  (null)  (1.8T)

Thanks.

This is very wrong... Don't try to issue any other ZFS commands. Use an MFS 10.1 bootable iso to boot and then issue:

# gpart list
# gpart show -l

Post the results here. Do not attempt to recover anything before asking. Keep in mind that any advice that you might get with commands that write stuff into your disks might result in data loss.

Cyberhide · Feb 12, 2015

Have you been able to resolve this?

I have the same exact situation with a FreeNAS box (9.1.1). I had a USB stick go bad, the machine kernel panic'd, and I had to do a fresh install on a new stick. Now I cant see my volume whatsoever...

I can see the disks:

Code:

[root@freenas] ~# camcontrol devlist
<Hitachi HDS723030ALA640 MKAOA5C0>  at scbus0 target 0 lun 0 (ada0,pass0)
<HUA723030ALA640 MKAOA870>         at scbus1 target 0 lun 0 (ada1,pass1)
<ST4000VN000-1H4168 SC46>          at scbus2 target 0 lun 0 (ada2,pass2)
<ST4000VN000-1H4168 SC46>          at scbus3 target 0 lun 0 (ada3,pass3)

I can see the ZFS partitions:

Code:

[root@freenas] ~# gpart show
=>        34  5860533101  ada0  GPT  (2.7T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338703     2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338703     2  freebsd-zfs  (2.7T)

=>        34  7814037101  ada2  GPT  (3.7T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  7809842696     2  freebsd-zfs  (3.7T)
  7814037128           7        - free -  (3.5k)

=>        34  7814037101  ada3  GPT  (3.7T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  7809842696     2  freebsd-zfs  (3.7T)
  7814037128           7        - free -  (3.5k)

But zpool status and zpool import do not work:

Code:

[root@freenas] ~# zpool status -v
no pools available
[root@freenas] ~# zpool import -a
[root@freenas] ~# zpool status -v
no pools available

I also checked the zdb info, and I see no labels (just like the OP) for any disk:

Code:

[root@freenas] ~# zdb -l /dev/ada0p2
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

-Chris

HarryE · Feb 12, 2015

It seems FreeNAS repartitioned your disks on install. That may be the reason you can't import the pool and zdb can't find the pool labels. You may try to find the string pool_guid somewhere on the disk. See above. Use combinations of dd, strings and grep to determine the position of "pool_guid". Use only one of the disks included in former pool. Do not write on those disks!

AndrewS · Jan 6, 2016

Reviving. I'm having this exact same issue. Had the boot media die, installed fresh onto new USB, and now my pools are invisible. Did the original poster ever resolve this? Has anybody else run into this? This has to be recoverable. The data is right there on the disks.

HarryE, I understand in principle what you're trying to relate. We find the pool_guid, then fix the partition table so the pool_guid is in the right place in respect to the beginning of the partition.

What I need is more specificity in how I determine the pool_guid. Is it as simple as dd if=<zfs partition> | strings | grep pool_guid? I've been running that against my disks for several minutes and haven't found anything matching that string. I would expect it to be pretty close to the beginning of the partition.

Help!

ZFS Need help to recover data from damaged ZFS pool

␢