Solved EFI partition - mount_msdosfs: /dev/mirror/p1: Invalid argument

DonK · Oct 10, 2022

In this case "clone" means "dump a copy of UFS data from mirror/gm0 to mirror/gm1. The commands shown above, and only the commands shown above, create the metadata. The metadata is created manually as shown above and not cloned from a dump.

Nothing other than what's shown above takes place from one reboot to the next reboot. In other words, one or more of the commands shown above is the only available "something" to corrupt the metadata.

And yes, there is a reason for the mirror to be corrupted. It's well known GPT doesn't work with mirrored drives.

Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme

... By default gmirror and the GPT partitioning scheme do not get along. This is because both GEOM (the provider for gmirror) and GPT write meta data at the end of the disk. ...

_martin · Oct 10, 2022

DonK said:
This is because both GEOM (the provider for gmirror) and GPT write meta data at the end of the disk. ...

End of its provider, not disk itself. And disk is one sector larger than the mirror provider. gmirror device doesn't see the physical end of the disk, GPT does. I had gmirrors on root for many years and never had any corruption of it. Only that itch that was explained. Then on FreeBSD 8 I migrated to zfs.

It's hard to troubleshoot like this without being able to execute commands and see the output more interactively. But I can share what I'd do:
a) make sure all disks are clean (though gpart destroy does that)
b) save a hexdump copy each step of all 4 disks so you are able to compare it later ; e.g
dd if=/dev/ada0 of=start_ada0 bs=512 count=32
do the same for end of the disk (diskinfo shows you how much sectors you have on the disk, use dd to skip over size-32) and do the same
c) do the same on gmirror devices, each step

DonK · Oct 10, 2022

As stated by me earlier, GPT corruption on mirrored HDDs is a well known problem:

FreeBSD Handbook 19.3.1. Metadata Issues

gmirror(8) stores one block of metadata at the end of the disk. As GPT partition schemes also store metadata at the end of the disk, mirroring entire GPT disks with gmirror(8) is not recommended. MBR partitioning is used here because it only stores a partition table at the start of the disk and does not conflict with the mirror metadata.

Here's the race condition mentioned by me earlier:

gmirror gm0 destroyed on shutdown; GPT corrupt

Using the last sector is not only flawed because it creates a race
condition, it's flawed in the assumption that you can always make
a geom part of a mirror by storing meta-data on the geom without
causing corruption.

avner · Oct 10, 2022

GEOM: mirror/gm1: the secondary GPT table is corrupt or invalid.

I think that the above message can be safely ignored. If the mirror is created from partitions then it is not a corruption; rather the secondary GPT table is hidden at the geom provider level. If the mirror is created from whole disk after creation of gpt partition table on the whole disk then yes last sector ( secondary partition table ) may be overwritten; but even so the disk will function normally just with no backup partition table. In your case you GPT partitioned the mirror not the raw disc. This too hides the secondary partition table; ie creates it in the second to last sector; again not a problem. TLDR, agree with what _martin has already stated.

avner · Oct 10, 2022

DonK said:
As stated by me earlier, GPT corruption on mirrored HDDs is a well known problem:

FreeBSD Handbook 19.3.1. Metadata Issues

gmirror(8) stores one block of metadata at the end of the disk. As GPT partition schemes also store metadata at the end of the disk, mirroring entire GPT disks with gmirror(8) is not recommended. MBR partitioning is used here because it only stores a partition table at the start of the disk and does not conflict with the mirror metadata.

Here's the race condition mentioned by me earlier:

gmirror gm0 destroyed on shutdown; GPT corrupt

Using the last sector is not only flawed because it creates a race
condition, it's flawed in the assumption that you can always make
a geom part of a mirror by storing meta-data on the geom without
causing corruption.

Please also note response by Pawel Jakub Dawidek here https://lists.freebsd.org/pipermail/freebsd-geom/2009-June/003594.html

_martin · Oct 11, 2022

DonK Well, I've to say I was wrong. I was convinced gmirror showing the size of the provider (-1 from raw disk) *including* its metadata. I rerun it again on new disks:

Code:

vmfbsd13(~)# diskinfo da{2,3}
da2    512    37580963840    73400320    0    0    4568    255    63
da3    512    37580963840    73400320    0    0    4568    255    63
vmfbsd13(~)#

vmfbsd13(~)# gmirror label -v gm0 da2 da3
Metadata value stored on da2.
Metadata value stored on da3.
Done.
vmfbsd13(~)#

vmfbsd13(~)# dd if=/dev/da2 skip=73400300 | (sleep 0.1 && hd)
20+0 records in
20+0 records out
10240 bytes transferred in 0.005475 secs (1870314 bytes/sec)
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002600  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
00002610  04 00 00 00 67 6d 30 00  9c 88 51 7f 8a 97 3c 63  |....gm0...Q...<c|
00002620  40 02 00 00 71 84 97 ce  2f 79 c3 60 02 00 00 00  |@...q.../y.`....|
00002630  00 01 00 00 00 00 00 10  00 00 02 00 fe ff bf 08  |................|
00002640  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
00002650  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002670  00 00 c0 08 00 00 00 4a  14 21 e8 95 a6 be 20 6a  |.......J.!.... j|
00002680  58 d6 49 db 8e c2 04 00  00 00 00 00 00 00 00 00  |X.I.............|
00002690  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002800
vmfbsd13(~)#

..and yeah, it's been put on last sector of the disk.

So it's not only annoying itch but as you said those two are fighting about last sector. I saw what you pasted and read the mailing list. I don't agree with the approach but that's irrelevant.
I remember I didn't have any problems but now I would not put wager on it. I did move to ZFS rather early on FreeBSD.

VladiBG · Oct 12, 2022

_martin you are not wrong this is LBA-1 on /dev/da0 which holds the metadata of gmirror.
When you create the GPT partition it will be placed inside the mirror provider which is /dev/mirror/gm0 and it's secondary GPT header will be before gmirror metadata at LBA-2 to LBA-34 of /dev/da0 or LBA-1 to LBA-33 on /dev/mirror/gm0

gpart create -s GPT /dev/mirror/gm0

note:
your stripesize of the disk is not recognized /reported/ in above post.

_martin · Oct 12, 2022

VladiBG I didn't even follow my advise. I should have created gpt header as you said and then compare the output. I do have excuse why I was so inconsistent but no excuses.

I'll do it again once I've more free time.
Those were virtual disks under vmware so maybe that's why.

Erichans · Oct 12, 2022

VladiBG said:
[...] When you create the GPT partition it will be placed inside the mirror provider which is /dev/mirror/gm0 and its secondary GPT header will be before gmirror metadata at LBA-2 to LBA-34 of /dev/da0 or LBA-1 to LBA-33 on /dev/mirror/gm0

gpart create -s GPT /dev/mirror/gm0

Yes, but I think there is tension between GEOM structures and GPT's disk layout. FreeBSD's GEOM structures (GEOM & GEOM Tutorial) allow for nested providers and AFAIK are unique. GPT (GUID Partition Table) has been made part of the UEFI spec and, from what I can tell does not take into account FreeBSD's "logical devices" such as FreeBSD's GEOM providers.

The UEFI spec (specifications overview) 5.3. GUID Partition Table (GPT) Disk Layout - 5.3.1. GPT overview:

[...]
Two GPT Header structures are stored on the device: the primary and the backup. The primary GPT Header must be located in LBA 1 (i.e., the second logical block), and the backup GPT Header must be located in the last LBA of the device.

In the context of the spec "the last LBA of the device" leaves no possibility other than that the backup header and its accompanying partition table should reside at the end of the physical device. Also noted by wblock@ here.

FreeBSD and its GEOM structures interpret these GPT rules (in relation to the backup header and partition table info) slightly different (bends the rules a bit if you prefer) in order to let GEOM structures coexist with the GPT layout and that seems a good practical approach.

When you already have a GEOM structure that has put meta data at the very end of the physical disk then, when you create—by FreeBSD means—a GPT partition inside that GEOM structure, you have a good chance to get things working.

You might be trying to mount such a disk with a non-FreeBSD OS or use other *NIX/other OS GPT disk-utils, or even utils from ports that do not take into account possible GEOM provider nesting—perhaps because they are ported from outside the FreeBSD domain. In that case, one should be extra careful to create, repair, modify or destroy GPT partitions.