ZFS The pool metadata is corrupted, how to get your data out of the corrupted zpool?

[freejlr]

I have to double check, but I am not sure, the /root home folder, where GPG keys are stored is that located under zroot/ROOT/default? Or does that come via zroot , the highest level ...

Correct the root is located in the dataset zroot/ROOT/default, all mount points that are not specified separately, such as zroot/var, will depend on and be located in zroot/ROOT/default.

zfs list -o name,mountpoint

 

[Relictrix]

Hi,

To summarize these are the steps, some of the actions you need to repeat for each dataset:

Source:
1. Start your machine in single boot modus
2. Choose shell
3. Set keyboard layout with kbdmap
4. Decrypt disk with geli: geli attach ada0p3
5. Set ip: ifconfig em1 inet 10.0.2.10 netmask 255.255.255.0
6. Next get object id of dataset to send to another server
zdb -d zroot/media/book -e

Destination:
1. Start the destination also in single boot modus
2. Choose shell
3. Set keyboard layout with kbdmap
4. Decrypt disk with geli: geli attach ada0p3
5. Import pool:
zpool import zroot
6. The destination server is 10.0.2.11:
ifconfig em1 inet 10.0.2.11 netmask 255.255.255.0
7. Destination listens and receive it in a new dataset:
nc -l 4242 | zfs receive zroot/media/book

Source:
1. Then I copy it to another machine:
zdb -B zroot/115478 -e | nc 10.0.2.11 -p 4242
2. When the command fails with a dump error, I think it means that this dataset is corrupt.

Then the destination will try to mount the new dataset but that will fail. Ignore that error.

When finished with all datasets reboot the destination and start normally. Then all datasets will be visible and mounted.

It's quiet time-consuming.

 

[Relictrix]

I do notice a difference in one dataset when comparing space on zfs versus du:

user1@vm1:/usr/home % zfs list zroot/usr/home
NAME             USED  AVAIL  REFER  MOUNTPOINT
zroot/usr/home  8.10G  2.93G  8.10G  /usr/home

user1@vm1:/usr/home % du -chxs *
327M    user1
327M    total
user1@vm1:/usr/home %

There are no reservations, no duplication, no hidden folders, ...

 

[freejlr]

I do notice a difference in one dataset when comparing space on zfs versus du:

user1@vm1:/usr/home % zfs list zroot/usr/home
NAME             USED  AVAIL  REFER  MOUNTPOINT
zroot/usr/home  8.10G  2.93G  8.10G  /usr/home

user1@vm1:/usr/home % du -chxs *
327M    user1
327M    total
user1@vm1:/usr/home %

There are no reservations, no duplication, no hidden folders, ...

What would be the output of the following command?

zfs list -o space zroot/usr/home

How many files are there in that datasheet?

 

[Relictrix]

Hi,

This is the output of the statement:

root@vm1:/usr/home # zfs list -o space zroot/usr/home
NAME            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot/usr/home  2.58G  8.10G        0B   8.10G             0B         0B
root@vm1:/usr/home #

NAME            PROPERTY              VALUE                     SOURCE
zroot/usr/home  type                  filesystem                -
zroot/usr/home  creation              Tue Aug 27 17:59 2024     -
zroot/usr/home  used                  8.10G                     -
zroot/usr/home  available             2.58G                     -
zroot/usr/home  referenced            8.10G                     -
zroot/usr/home  compressratio         1.56x                     -
zroot/usr/home  mounted               no                        -
zroot/usr/home  quota                 none                      default
zroot/usr/home  reservation           none                      default
zroot/usr/home  recordsize            128K                      default
zroot/usr/home  mountpoint            /usr/home                 inherited from zroot/usr
zroot/usr/home  sharenfs              off                       default
zroot/usr/home  checksum              on                        default
zroot/usr/home  compression           lz4                       inherited from zroot
zroot/usr/home  atime                 off                       inherited from zroot
zroot/usr/home  devices               on                        default
zroot/usr/home  exec                  on                        default
zroot/usr/home  setuid                on                        default
zroot/usr/home  readonly              off                       default
zroot/usr/home  jailed                off                       default
zroot/usr/home  snapdir               hidden                    default
zroot/usr/home  aclmode               discard                   default
zroot/usr/home  aclinherit            restricted                default
zroot/usr/home  createtxg             7906                      -
zroot/usr/home  canmount              on                        default
zroot/usr/home  xattr                 on                        default
zroot/usr/home  copies                1                         default
zroot/usr/home  version               5                         -
zroot/usr/home  utf8only              off                       -
zroot/usr/home  normalization         none                      -
zroot/usr/home  casesensitivity       sensitive                 -
zroot/usr/home  vscan                 off                       default
zroot/usr/home  nbmand                off                       default
zroot/usr/home  sharesmb              off                       default
zroot/usr/home  refquota              none                      default
zroot/usr/home  refreservation        none                      default
zroot/usr/home  guid                  7845144682796008424       -
zroot/usr/home  primarycache          all                       default
zroot/usr/home  secondarycache        all                       default
zroot/usr/home  usedbysnapshots       0B                        -
zroot/usr/home  usedbydataset         8.10G                     -
zroot/usr/home  usedbychildren        0B                        -
zroot/usr/home  usedbyrefreservation  0B                        -
zroot/usr/home  logbias               latency                   default
zroot/usr/home  objsetid              4363                      -
zroot/usr/home  dedup                 off                       default
zroot/usr/home  mlslabel              none                      default
zroot/usr/home  sync                  standard                  default
zroot/usr/home  dnodesize             legacy                    default
zroot/usr/home  refcompressratio      1.56x                     -
zroot/usr/home  written               8.10G                     -
zroot/usr/home  logicalused           11.3G                     -
zroot/usr/home  logicalreferenced     11.3G                     -
zroot/usr/home  volmode               default                   default
zroot/usr/home  filesystem_limit      none                      default
zroot/usr/home  snapshot_limit        none                      default
zroot/usr/home  filesystem_count      none                      default
zroot/usr/home  snapshot_count        none                      default
zroot/usr/home  snapdev               hidden                    default
zroot/usr/home  acltype               nfsv4                     default
zroot/usr/home  context               none                      default
zroot/usr/home  fscontext             none                      default
zroot/usr/home  defcontext            none                      default
zroot/usr/home  rootcontext           none                      default
zroot/usr/home  relatime              on                        default
zroot/usr/home  redundant_metadata    all                       default
zroot/usr/home  overlay               on                        default
zroot/usr/home  encryption            off                       default
zroot/usr/home  keylocation           none                      default
zroot/usr/home  keyformat             none                      default
zroot/usr/home  pbkdf2iters           0                         default
zroot/usr/home  special_small_blocks  0                         default
zroot/usr/home  snapshots_changed     Thu Sep  5 13:36:30 2024  -
zroot/usr/home  prefetch              all                       default

Here recursive from zroot/usr:

root@vm1:/usr/home # zfs list -o space -r zroot/usr
NAME             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot/usr        2.58G  13.1G        0B    117K             0B      13.1G
zroot/usr/home   2.58G  8.10G        0B   8.10G             0B         0B
zroot/usr/ports  2.58G  3.33G        0B   3.33G             0B         0B
zroot/usr/src    2.58G  1.71G     1.26M   1.71G             0B         0B
root@vm1:/usr/home #

The mountpoint is inherited from zroot/usr I see , has that got something to do with it?

 

[hex_messer]

Hi!
I'm trapped into same situation: FreeBSD on zfs inside Virtualbox. Nothing worthy. But the fact is surpising for me! At first that zfs can die without any hope to recovery (I not use a lot of setups, so my skills are poor, yes all my setups without duplication), and at second that there can be hardware I/O disk errors only inside Virtualbox!

Yep, I have always believe that inside virtualbox we will never see any hard drive errors like on real hw, but YES, they can be!, as someone proposed on first page. After hang up of the host while running virtualbox guest I got unbootable guest system with the same (yes, unrecoverable in my case, as in yours) error. And when I do check the reliability (with dd) of the drive (guest drive), I've surprisingly got I/O errors, not only on FreeBSD, but with linux guest also. On the host system file can be read without any errors as file (also dd for example), but copying virtual drive by Virtualbos storage management tools fails.

May be it be useful: virtualbox have no instruments to check consistency of VDI, i reveal error only by copying/converting process. But qemu-img convert this corrupted VDI file to another (i choose VMDK) without any errors, and inside quest I at least see no hardware I/O errors (also, for example, using dd). But zfs pool still cannot be recovered....