RAID lost after Upgrading

G

Gábor Fülöp

Hi Guys!

First up, sorry for my english ;)
I was trying to update from 10.3 to 11.1, and right after the first reboot, the system doesnt boot from the mirror. (Maybe the bootloader changes and the geom-mirror.ko doesnt loaded.)
I didn't think throught what i'm doing, just was happy to boot from ada1.
That was about a week ago, but i want prevent data loss.

Code: 
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <KINGSTON SV300S37A60G 505ABBF0> ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B723203EB7E
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 57241MB (117231408 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <KINGSTON SV300S37A60G 505ABBF0> ATA8-ACS SATA 3.x device
ada1: Serial Number 50026B723203EFE9
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 57241MB (117231408 512 byte sectors)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <TOSHIBA DT01ACA100 MS2OA750> ATA8-ACS SATA 3.x device
ada2: Serial Number Z628VSENS
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD20PURX-64P6ZY0 80.00A80> ACS-2 ATA SATA 3.x device
ada3: Serial Number WD-WCC4M1KVEHXS
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors)
ada3: quirks=0x1<4K>
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD1001FALS-00J7B1 05.00K05> ATA8-ACS SATA 2.x device
ada4: Serial Number WD-WMATV2099508
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 953869MB (1953525168 512 byte sectors)
Trying to mount root from ufs:/dev/mirror/root [rw]...
GEOM: ada1: the secondary GPT table is corrupt or invalid.
GEOM: ada1: using the primary only -- recovery suggested.
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
GEOM: diskid/DISK-50026B723203EFE9: the secondary GPT table is corrupt or invalid.
GEOM: diskid/DISK-50026B723203EFE9: using the primary only -- recovery suggested.
Root mount waiting for: usbus4
Root mount waiting for: usbus4
Root mount waiting for: usbus4
uhub2: 8 ports with 8 removable, self powered
Root mount waiting for: usbus4
mountroot: waiting for device /dev/mirror/root...
ugen0.2: <Motorola USB Keyboard  Mouse> at usbus0
ukbd0 on uhub4
ukbd0: <Keyboard> on usbus0
kbd2 at ukbd0
Mounting from ufs:/dev/mirror/root failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/mirror/root
  vfs.root.mountfrom.options=rw

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot> Trying to mount root from ufs:/dev/ada0p3 []...
g_vfs_done():ada0p1[READ(offset=262144, length=8192)]error = 5
Mounting from ufs:/dev/ada0p3 failed with error 5.

mountroot> random: unblocking device.
Invalid file system specification.

mountroot> Trying to mount root from ufs:/dev/ada1p3 []...
GEOM: diskid/DISK-50026B723203EFE9: the secondary GPT table is corrupt or invalid.
GEOM: diskid/DISK-50026B723203EFE9: using the primary only -- recovery suggested.
ums0 on uhub4
ums0: <Mouse> on usbus0
ums0: 5 buttons and [XYZ] coordinates ID=1
ums0: 5 buttons and [Z] coordinates ID=4
em0: link state changed to UP
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_MIRROR: Cannot open consumer ada1 (error=1).
GEOM_MIRROR: Cannot add disk ada1 to gm0 (error=1).
GEOM_MIRROR: Device gm0 destroyed.
GEOM_MIRROR: Cannot open consumer ada1p3 (error=1).
GEOM_MIRROR: Cannot add disk ada1p3 to root (error=1).
GEOM_MIRROR: Device root destroyed.
GEOM_MIRROR: Device mirror/swap launched (2/2).
GEOM_MIRROR: Device mirror/boot launched (2/2).
GEOM_MIRROR: Force device root start due to timeout.
GEOM_MIRROR: Device mirror/root launched (1/2).
GEOM_NOP: Device gzero.nop created.

[CODE][root@xxx /var/log]# camcontrol devlist
<KINGSTON SV300S37A60G 505ABBF0>   at scbus0 target 0 lun 0 (ada0,pass0)
<KINGSTON SV300S37A60G 505ABBF0>   at scbus1 target 0 lun 0 (ada1,pass1)
<TOSHIBA DT01ACA100 MS2OA750>      at scbus2 target 0 lun 0 (ada2,pass2)
<WDC WD20PURX-64P6ZY0 80.00A80>    at scbus3 target 0 lun 0 (ada3,pass3)
<WDC WD1001FALS-00J7B1 05.00K05>   at scbus4 target 0 lun 0 (ada4,pass4)

Code: 
[root@xxx /var/log]# gpart show
=>       34  117231341  diskid/DISK-50026B723203EB7E  GPT  (56G)
         34        128                             1  freebsd-boot  (64K)
        162    2097152                             2  freebsd-swap  (1.0G)
    2097314  115134061                             3  freebsd-ufs  (55G)

=>       34  117231341  ada1  GPT  (56G) [CORRUPT]
         34        128     1  freebsd-boot  (64K)
        162    2097152     2  freebsd-swap  (1.0G)
    2097314  115134061     3  freebsd-ufs  (55G)

=>        34  1953525101  ada2  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048  1953521664     1  freebsd-ufs  (932G)
  1953523712        1423        - free -  (712K)

=>        34  3907029101  ada3  GPT  (1.8T)
          34           6        - free -  (3.0K)
          40  3907029088     1  freebsd-ufs  (1.8T)
  3907029128           7        - free -  (3.5K)

=>        63  1953525105  ada4  MBR  (932G)
          63  1258290369     1  freebsd  [active]  (600G)
  1258290432   419429808     2  freebsd  [active]  (200G)
  1677720240   275804928     3  freebsd  [active]  (132G)

Code: 
[root@xxx /home/xxx]# gpart show ada0
gpart: No such geom: ada0.

So, the question is: Can i rebuild the mirror without reboot? If yes, how?
Thank you !
 
How did you perform the upgrade? Did you build the system yourself or did you use freebsd-update? Also noteworthy: 11.1 has met it's EOL, the current version is 11.2 right now.

Anyway, what does # gmirror status and/or # gmirror list tell you, can it actually detect the mirror at all?
 
Code: 
[root@xxx /home/xxx]# uname -a
FreeBSD polymix.polymix.hu 11.1-RELEASE-p11 FreeBSD 11.1-RELEASE-p11 #0: Thu Jun 21 03:46:08 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Code: 
[root@xxx /home/xxx]# gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  ada1p2 (ACTIVE)
                       diskid/DISK-50026B723203EB7Ep2 (ACTIVE)
mirror/boot  COMPLETE  ada1p1 (ACTIVE)
                       diskid/DISK-50026B723203EB7Ep1 (ACTIVE)
mirror/root  DEGRADED  diskid/DISK-50026B723203EB7Ep3 (ACTIVE)

Code: 
[root@xxx/home/xxx]# gmirror list
Geom name: swap
State: COMPLETE
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 2858247452
Type: AUTOMATIC
Providers:
1. Name: mirror/swap
   Mediasize: 1073741312 (1.0G)
   Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: ada1p2
   Mediasize: 1073741824 (1.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 82944
   Mode: r1w1e1
   State: ACTIVE
   Priority: 1
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 1443635479
2. Name: diskid/DISK-50026B723203EB7Ep2
   Mediasize: 1073741824 (1.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 82944
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 1892787372

Geom name: boot
State: COMPLETE
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3709005870
Type: AUTOMATIC
Providers:
1. Name: mirror/boot
   Mediasize: 65024 (64K)
   Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: ada1p1
   Mediasize: 65536 (64K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r1w1e1
   State: ACTIVE
   Priority: 1
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 1125630478
2. Name: diskid/DISK-50026B723203EB7Ep1
   Mediasize: 65536 (64K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 1902698882

Geom name: root
State: DEGRADED
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 1801314001
Type: AUTOMATIC
Providers:
1. Name: mirror/root
   Mediasize: 58948638720 (55G)
   Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: diskid/DISK-50026B723203EB7Ep3
   Mediasize: 58948639232 (55G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1073824768
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 661682007

The gpart recovery doesnt work, as ada0 doesnt exists.
 
Your problem isn't so much with ada0 but ada1. I ignored it at first but notice how gpart marks it as corrupted in your first message? And that status got confirmed because ada0, though renamed, is still fully active within the RAID.

Although it is theoretically doable to put the freebsd-boot in RAID the facts are different. Disks usually aren't fully the same so they normally require customized boot code. For what's it worth I never try to mirror freebsd-boot but I simply re-run gpart bootcode for every bootdisk, never failed me so far.

Anyway, I strongly suggest you back up your data and re-create the mirror, but only limit it to root and swap. I'd also do some hardware tests (SMART?) on ada1, maybe even consider replacing it.
 
GPT store it's backup metadata on the end of the disk where the gmirror also store one block of metadata. That's why it's not recommended to use GPT with gmirror. Rebuilding the GPT will cause your gmirror metadata to be overwrite and the mirror will fail. You can still use gmirror but instead of providing it with entire disk you can mirror only selected partitions. If you want to mirror the entire disk then use MBR partition scheme.

If you have intel chipset it would be better to use graid insted of gmirror.
 
VladiBG said:
GPT store it's backup metadata on the end of the disk where the gmirror also store one block of metadata. That's why it's not recommended to use GPT with gmirror.
Click to expand...
This argument is bogus especially within the context of the OP. gmirror stores its meta data at the end of the provider, which in this case are individual partitions. So there's no risk, what so ever, for any overlap.

It could become a different story if you'd let gmirror use the entire disk, but even then it's debatable.
 
https://www.freebsd.org/doc/handbook/geom-mirror.html

gmirror(8) stores one block of metadata at the end of the disk. Because GPT partition schemes also store metadata at the end of the disk, mirroring entire GPT disks with gmirror(8) is not recommended. MBR partitioning is used here because it only stores a partition table at the start of the disk and does not conflict with the mirror metadata.
Click to expand...

I agree it store it in the provider's last sector. If the provider is the entire disk then it cause the conflict with the GPT metadata.

From the OP there's this dmesg
GEOM: ada1: the secondary GPT table is corrupt or invalid.
Click to expand...

Which tells me that the backup GPT metadata which is stored at the end of the disk is corrupted. From there's is my conclusion. It also possible to be caused of the bad sector at the end of the disk which is unlikely as those are SSDs.
 
Last edited:
Thank you guys for informative answers!
I checked the content of the disks and that was true, the mirror was working.. So because last time I booted from ada1, now i'm trying to rebuild the mirror without the swap partition, but i have another issue here:

Code: 
[root@xxx /backup2/fs]# gpart restore -l /dev/ada0 < table.ada1
gpart: start '34': Invalid argument

I'm trying to do step by step the partitioning too, but it starts always from 40.sector:

Code: 
[root@xxx /backup2/fs]# gpart create -s gpt ada0
ada0 created
[root@xxx /backup2/fs]# gpart add -s 64k -t freebsd-boot -l boot0 ada0
ada0p1 added
[root@xxx /backup2/fs]# gpart show ada0
=>       40  117231328  ada0  GPT  (56G)
         40        128     1  freebsd-boot  (64K)
        168  117231200        - free -  (56G)

Code: 
[root@xxx /backup2/fs]# gpart add -b 34 -s 64k -t freebsd-boot -l boot0 ada0
gpart: start '34': Invalid argument

Maybe is that a FreeBSD version issue?
 
The GPT partition table is 34 blocks (34 * 512 bytes = 17 kB). It's doing this because the disk need to be with 4k alignment. That's why it start at LBA 40 ( 40 * 512 bytes = 20480 bytes / 4096 bytes= 5)

https://en.wikipedia.org/wiki/GUID_Partition_Table#/media/File:GUID_Partition_Table_Scheme.svg

Edit:
If your partition start at the LBA 34 it will be not alignment to 4k (34 * 512 bytes = 17408 / 4096 = 4,25 which is not whole number) and your disk performance will suffer.
 
Then how was that possible to partitioning a few years ago like that?

Code: 
=>       34  117231341  ada1  GPT  (56G)
         34        128     1  freebsd-boot  (64K)
        162    2097152     2  freebsd-swap  (1.0G)
    2097314  115134061     3  freebsd-ufs  (55G)
 
Thanks Vladi!

It's wonderful :D It's not enough lost my mirror, i dont have any chance to rebuild it.
So what i hopefully can do, is a fresh install (11.2) to ada0, copy there the datas, and after that get rid of this ada1 scheme.

How do you see that?
 
Yes it would be easy with fresh install. But maybe instead of using gmirror you can go for ZFS or if you Motherboard has intel ICHR you can use graid to create RAID1 and then restore all data.
 
You must log in or register to reply here.
Back
Top