UFS Trouble mounting UFS2 after breaking gmirror (potential data loss)

Hi all,

I broke a gmirror that used to consist of two 12TB disks (ada2 + ada4). The UFS2 filesystem lived on /dev/mirror/mirror0.
After tearing down the mirror, the system recreated it once on reboot, I removed the metadata again, and now:
  • ada2p1 still shows a valid UFS2 superblock (file -s looks normal, clean flag set)
  • but mounting fails with:
    mount: /dev/ada2p1: Input/output error
  • fsck_ufs -n can read the primary superblock but gives I/O errors trying to check it
  • alternate superblocks are unreadable — fsck just prints their locations and exits
  • geom mirror list shows no mirror now
  • gpart show ada2 shows a normal 11TB freebsd-ufs partition
Important detail:
ada4p1 (the other half of the mirror) was overwritten as ZFS (freebsd-zfs partition), so only ada2p1 remains with UFS data.

It looks like the UFS superblock is intact, but the cylinder group / alternate superblock area is no longer readable, possibly due to geometry changes during the mirror break + a growfs attempt.
Questions:
  • Is it possible to regenerate alternate superblocks or CG metadata from the valid primary?
  • Is there a method to force UFS to walk the disk even if some CG headers are unreadable?
  • did I lose all of my data?
Thanks for any guidance.
 
Here's where I broke things:
Code:
Nov 20 19:46:56 dev kernel: GEOM_MIRROR: Device mirror0: provider ada4p1 destroyed.
Nov 20 19:52:11 dev kernel: GEOM_MIRROR: Device mirror0: provider destroyed.
Nov 20 19:52:11 dev kernel: GEOM_MIRROR: Device mirror0 destroyed.
Nov 20 19:55:02 dev kernel: GEOM_CONCAT: Device raidlarge created (id=3915430123).
Nov 20 19:55:02 dev kernel: GEOM_CONCAT: Disk ada2p1 attached to raidlarge.
Nov 20 19:55:02 dev kernel: GEOM_CONCAT: Disk ada4p1 attached to raidlarge.
Nov 20 19:55:02 dev kernel: GEOM_CONCAT: Device concat/raidlarge activated.
Nov 20 19:59:33 dev kernel: GEOM_CONCAT: Device raidlarge already configured.
Nov 20 20:02:19 dev kernel: GEOM_MIRROR: Device mirror/mirror0 launched (1/1).
Nov 20 20:02:19 dev kernel: GEOM_CONCAT: Disk ada2p1 removed from raidlarge.
Nov 20 20:02:19 dev kernel: GEOM_CONCAT: Device concat/raidlarge deactivated.
Nov 20 20:07:04 dev kernel: GEOM_CONCAT: Device raidlarge already configured.
Nov 20 20:07:04 dev syslogd: last message repeated 1 times
Nov 20 20:22:53 dev openvpn[1156]: TLS Error: cannot locate HMAC in incoming packet from [AF_INET6]::ffff:199.195.248.205:37983
Nov 20 20:22:53 dev openvpn[1156]: TLS Error: cannot locate HMAC in incoming packet from [AF_INET6]::ffff:199.195.248.205:37984
Nov 20 20:27:42 dev kernel: GEOM_CONCAT: Disk ada4p1 removed from raidlarge.
Nov 20 20:27:42 dev kernel: GEOM_CONCAT: Device raidlarge destroyed.
I ran out of space on my mirrored raid array, so i thought i would break that apart and use gconcat to have one big drive. Mistakes were made.
 
How did you clear the metadata from the gmirror?
Did you remove the geom_mirror_load="YES" from /boot/loader.conf after you destroy the mirror?
 
Code:
cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
fusefs_load="YES"
geom_mirror_load="YES"
vmm_load="YES"
nmdm_load="YES"
No, I did not.
 
Back
Top