ZFS zohome - My Pool With All My Stuff - Disappeared

Geezer · Sunday at 1:59 PM

On my main machines, I have four NVMEs, in two mirrors, one for the system, and the other for home.

There was just a power outage. The machine shutdown, it has a UPS and I am running nut. I did shutdown properly, I watched it.

Now I have brought it up, the pool for home has disappeared.

I can still see the NVME devices:
ls -1 /dev/nv*

Code:

/dev/nvd0
/dev/nvd0p1
/dev/nvd0p2
/dev/nvd0p3
/dev/nvd1
/dev/nvd1p1
/dev/nvd1p2
/dev/nvd1p3
/dev/nvd2
/dev/nvd3
/dev/nvme0
/dev/nvme0ns1
/dev/nvme1
/dev/nvme1ns1
/dev/nvme2
/dev/nvme2ns1
/dev/nvme3
/dev/nvme3ns1

But the zpool is not there:
zpool list

Code:

NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zoroot   308G  34.8G   273G        -         -    15%    11%  1.00x    ONLINE  -

There should be a pool 'zohome'.

I have tried zpool import
and get

Code:

no pools available to import

I can see the disks are still there:
geom disk list

Code:

Geom name: nda0                                                                                                                          
Providers:                                                                                                                               
1. Name: nda0                                                                                                                            
   Mediasize: 400088457216 (373G)                                                                                                        
   Sectorsize: 512                                                                                                                       
   Mode: r3w3e5                                                                                                                          
   descr: Micron_7300_MTFDHBA400TDG                                                                                                      
   lunid: 000000000000000100a0750128121cff                                                                                               
   ident: 200128121CFF                                                                                                                   
   rotationrate: 0                                                                                                                       
   fwsectors: 0                                                                                                                          
   fwheads: 0                                                                                                                            
                                                                                                                                         
Geom name: nda1                                                                                                                          
Providers:                                                                                                                               
1. Name: nda1                                                                                                                            
   Mediasize: 400088457216 (373G)                                                                                                        
   Sectorsize: 512                                                                                                                       
   Mode: r2w2e3                                                                                                                          
   descr: Micron_7300_MTFDHBA400TDG                                                                                                      
   lunid: 000000000000000100a0750128126c3a                                                                                               
   ident: 200128126C3A                                                                                                                   
   rotationrate: 0                                                                                                                       
   fwsectors: 0                                                                                       
   fwheads: 0                                                                                         
                       
Geom name: nda2                                                                                       
Providers:                                                                                            
1. Name: nda2                                                                                         
   Mediasize: 1073741824 (1.0G)                                                                       
   Sectorsize: 512                                                                                    
   Mode: r0w0e0                                                                                       
   descr: SAMSUNG MZ1LB1T9HALS-00007                                                                  
   lunid: 343336304e7154820025384100000001                                                            
   ident: S436NA0N715482                                                                              
   rotationrate: 0                                                                                    
   fwsectors: 0                                                                                       
   fwheads: 0                                                                                         

Geom name: nda3                   
Providers:                        
1. Name: nda3                     
   Mediasize: 1073741824 (1.0G)                                     
   Sectorsize: 512                
   Mode: r0w0e0                   
   descr: SAMSUNG MZ1LB1T9HALS-00007                                
   lunid: 343336304e7155520025384100000001                          
   ident: S436NA0N715552          
   rotationrate: 0                
   fwsectors: 0                   
   fwheads: 0

What could have happened to my pool?

Is it still there on the two mirrored drives nda2 and nda3?

Most importantly, how can I get it back?

W.hâ/t · Sunday at 2:08 PM

You might need to import using its id, which will be listed if the following command fails, zpool-import(8).

Code:

# zpool import -d device

Geezer · Sunday at 2:12 PM

Additionally, zdb -l nvd2 and zdb -l nvd3 both give

Code:

failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

hexdump -C /dev/nvd2] and [cmd]hexdump -C /dev/nvd3

Code:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
40000000

Geezer · Sunday at 2:16 PM

W.hâ/t said:
You might need to import using its id, which will be listed if the following command fails, zpool-import(8).

Code:

# zpool import -d device

zpool import -d /dev/nda2 -d /dev/nda3

Code:

no pools available to import

Geezer · Sunday at 2:34 PM

Looking closer at geom disk list, it gives the size for nda2 and nda3 as Mediasize: 1073741824 (1.0G) .

This should be 2TB, not 1GB!

tingo · Sunday at 2:36 PM

Are the SSD's old?

Geezer · Sunday at 2:38 PM

tingo said:
Are the SSD's old?

Four years.

Could they both have failed, at once?

CeXP1917 · Sunday at 2:38 PM

Geezer said:
Mediasize: 1073741824 (1.0G)

Firmware crash?

//Gold rule: dont use Samsung and/or Seagate without backup.

Emrion · Sunday at 2:39 PM

What's about gpart show and zpool status?

W.hâ/t · Sunday at 2:40 PM

Geezer said:
Looking closer at geom disk list, it gives the size for nda2 and nda3 as Mediasize: 1073741824 (1.0G) .

This should be 2TB, not 1GB!

Indeed, the disks are empty. Looks like you've lost the data on those.

Maybe you'll find more info with camcontrol(8).

Geezer · Sunday at 2:40 PM

CeXP1917 said:
Firmware crash?

//Gold rule: dont use Samsung and/or Seagate without backup.

I do have a backup.

tingo · Sunday at 2:41 PM

And why does the drives show up as "ndaX" in geom disk list instead of "nvdX"?

Geezer · Sunday at 2:42 PM

Emrion said:
What's about gpart show and zpool status?

gpart show

Code:

=>       40  781422688  nda0  GPT  (373G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1.0M)
     534528  134217728     2  freebsd-swap  (64G)
  134752256  646670336     3  freebsd-zfs  (308G)
  781422592        136        - free -  (68K)

=>       40  781422688  nda1  GPT  (373G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1.0M)
     534528  134217728     2  freebsd-swap  (64G)
  134752256  646670336     3  freebsd-zfs  (308G)
  781422592        136        - free -  (68K)

gpart status

Code:

  Name  Status  Components
nda0p1      OK  nda0
nda0p2      OK  nda0
nda0p3      OK  nda0
nda1p1      OK  nda1
nda1p2      OK  nda1
nda1p3      OK  nda1

Emrion · Sunday at 2:43 PM

zpool status not gpart.

Seems two of your four disks have disappeared.

Geezer · Sunday at 2:44 PM

tingo said:
And why does the drives show up as "ndaX" in geom disk list instead of "nvdX"?

Don't know.

Geezer · Sunday at 2:46 PM

Emrion said:
zpool status not gpart.

zpool status

Code:

  pool: zoroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:13 with 0 errors on Sat Oct 11 03:05:33 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zoroot      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            nda0p3  ONLINE       0     0     0
            nda1p3  ONLINE       0     0     0

errors: No known data errors

No sign of zohome.

Emrion · Sunday at 2:52 PM

tingo said:
And why does the drives show up as "ndaX" in geom disk list instead of "nvdX"?

Ones are symlinks of others since a change of driver in the 13 or 14 era IIRC.
See https://forums.freebsd.org/threads/loader-needs-to-be-updated-14-stable.94857/page-3#post-708223

CeXP1917 · Sunday at 2:54 PM

Geezer said:
I do have a backup.

Good, so its safe to look at this ssd on other PC with distros for repair works, to exclude problems with PSU and other components on this machine. Firmware crash on 2 disks at same time is strange situation.

Geezer · Sunday at 3:00 PM

Emrion said:
Seems two of your four disks have disappeared.

Well, they are still there in geom disk list

covacat · Sunday at 3:00 PM

they. could very well fail at the same time
see https://secureteam.co.uk/2019/12/05/integer-overflow-flaw-hits-hp-ssd/

nxjoseph · Sunday at 3:01 PM

Geezer said:
Well, they are still there in geom disk list

but not in gpart show, also partitions are missing too. Is there any clue on dmesg(8)?

Geezer · Sunday at 3:02 PM

nxjoseph said:
but not in gpart show, also partitions are missing too.

There were no partitions. Used whole disks.

Geezer · Sunday at 3:04 PM

covacat said:
they. could very well fail at the same time
see https://secureteam.co.uk/2019/12/05/integer-overflow-flaw-hits-hp-ssd/

So much for redundancy in mirrors.

Actually, the drives are passed 32768 hours, and were both at about 39000 hours.

Emrion · Sunday at 3:07 PM

Geezer said:
Well, they are still there in geom disk list

Alas, I don't have answer to that. I just can imagine some damages on the partition tables of these disks, but I may be wrong.
This case is strange. Something - at least - escapes us.

Geezer · Sunday at 3:12 PM

Tomorrow (cos it is bloody late here and I am tired), I will take one of the drives out and put it in my gmktek, boot from a memstick and try and see if anything is there. That will tell me if the drives have failed, or the host machine.

I am still interested in all suggestions.