UFS gmirror synchronisation

fabric · Friday at 2:07 AM

Hi,

I have been experimenting a little with 2 external USB drives and gmirror. Placing the gmirror on the disk and then creating a single partition and also gmirror’ing only a smaller partition set.

I have noticed when I test its resilience by changing a small file on only one of the USB drives that when I reinsert the removed drive it rebuilds in entirety. Clearly, this works well, but is it meant to have some instrumentation to know only to rebuild the affected blocks or is it by design meant to rebuild the entire disk / partition regardless ?

Many thanks for any additional insight.

gpw928 · Friday at 4:11 AM

gmirror(8) has no knowledge of the underlying file system and keeps all its (very limited) metadata on the last sector of the mirror. So it's only option when one side of a mirror gets out of sync is to re-silver the entire mirror.

If the sync times are an issue, consider mirroring multiple (smaller) partitions, or use ZFS which has the advantage of knowing about the underlying hardware (well, provider) and the file system.

fabric · Friday at 1:58 PM

Thank you for the prompt reply, and language, you put that really well. As it turns out I am using ZFS on the main drive; the use of gmirror is for an off-board backup and I wanted it to be simple. I’ll try using a few smaller partitions. Also is it still recommended not to use GPT as the metadata clashes at the end of the disk? I‘m doing a before /after compare of hexdump for the last 1M of the disk to check for myself but it’s a bit inconclusive at this time.

Many thanks

gpw928 · Friday at 10:24 PM

fabric said:
Also is it still recommended not to use GPT as the metadata clashes at the end of the disk?

The short answer is that I still don't know. Sorry. I have migrated to mostly using ZFS, and these days only use gmirror for making swap partitions.

There's also a problem with software-mirroring an EFI partition. I'm still firmly of the view that it should not be software mirror'd with things like gmirror, as the EFI contents can be changed without the knowledge of the over-lying operating system (from the stand-alone shell-like command). Hardware RAID is usually OK, because the under-lying hardware RAID controller maintains integrity.

Jose · Saturday at 12:52 AM

fabric said:
Also is it still recommended not to use GPT as the metadata clashes at the end of the disk?

Yes.

Chapter 21. GEOM: Modular Disk Transformation Framework

In FreeBSD, the GEOM framework permits access and control to classes, such as Master Boot Records and BSD labels, through the use of providers, or the disk devices in /dev.

docs.freebsd.org

FWIW, I also had to have GEOM “re-taste” the partitions

Code:

true > /dev/ada0
true > /dev/ada1

That tip is from here:

HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme - A-Team Systems

HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme

www.ateamsystems.com

This was several years ago, though. I would use ZFS mirroring if I was going to set up that server today.

T-Daemon · Saturday at 10:36 AM

fabric said:
Also is it still recommended not to use GPT as the metadata clashes at the end of the disk?

Jose said:
Yes.

Chapter 21. GEOM: Modular Disk Transformation Framework

In FreeBSD, the GEOM framework permits access and control to classes, such as Master Boot Records and BSD labels, through the use of providers, or the disk devices in /dev.

docs.freebsd.org

I've notice many times that the FreeBSD documentation is not accurate (or outdated), which seems be the case with gmirror(8) overwriting GPT secondary table on the last sector of a disk as well.

The manual gmirror(8) states

Rich (BB code):

DESCRIPTION
     [...]
     The gmirror utility uses on-disk metadata (stored in the provider's last
     sector) to store all needed information. [...]

It says "provider's last sector", not disks. To my understanding a "provider" can be a disk or a partition.

I don't know on which source the author of the article bases his statement, but the mentioned FreeBSD 9 -> 11 versions, according to the manuals have all the same reference to a provider, not disk.

gmirror(8) appeared first in FreeBSD 5.3, it's manual also refers to a provider, not disk.

I think that "gmirror(8) metadata is written on the last sector of a disk" is a myth, a myth which made it in the handbook, took over unverified by the author of the chapter https://docs.freebsd.org/en/books/handbook/geom/#geom-mirror-metadata.

I did some (superficial) searching in src/sys/geom/mirror/, but didn't figure out how the metadata on which sector is handled.

So I tested in a VM and the results support my interpretation. gmirror(8) metadata does not overwrite GPT secondary table on the last sector of the disk.

First, gpart(8) show doesn't show the partition table corrupted after creating a gmirror(8), which would be the case if the secondary GPT table was overwritten, second, hexdump(1) confirms.

Test case 1:

Code:

=>      40  12582832  ada0  GPT  (6.0G)
        40      2008        - free -  (1.0M)
      2048  12578816     1  freebsd-ufs  (6.0G)
  12580864      2008        - free -  (1.0M)

=>      40  12582832  ada1  GPT  (6.0G)
        40      2008        - free -  (1.0M)
      2048  12578816     1  freebsd-ufs  (6.0G)
  12580864      2008        - free -  (1.0M)

hexdump(1) of ada0 after creating GPT table, showing primary and secondary GPT table

Rich (BB code):

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff bf 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  84 db 2a f6 00 00 00 00  01 00 00 00 00 00 00 00  |..*.............|
00000220  ff ff bf 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000230  d7 ff bf 00 00 00 00 00  54 a5 8f 32 e6 35 ef 11  |........T..2.5..|
00000240  b7 ac 08 00 27 c4 37 c5  02 00 00 00 00 00 00 00  |....'.7.........|
00000250  80 00 00 00 80 00 00 00  b5 4a 6b 97 00 00 00 00  |.........Jk.....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000410  b4 2e 44 61 e6 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..Da.5......'.7.|
00000420  00 08 00 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe10  b4 2e 44 61 e6 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..Da.5......'.7.|
17fffbe20  00 08 00 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
17fffbe30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
17ffffe10  28 e5 09 b4 00 00 00 00  ff ff bf 00 00 00 00 00  |(...............|
17ffffe20  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
17ffffe30  d7 ff bf 00 00 00 00 00  54 a5 8f 32 e6 35 ef 11  |........T..2.5..|
17ffffe40  b7 ac 08 00 27 c4 37 c5  df ff bf 00 00 00 00 00  |....'.7.........|
17ffffe50  80 00 00 00 80 00 00 00  b5 4a 6b 97 00 00 00 00  |.........Jk.....|
17ffffe60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

hexdump(1) of ada0 after creating a ada0p1/ada1p1 gmirror(8), label "ufsmir", primary and secondary GPT table highlighted

Rich (BB code):

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff bf 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  84 db 2a f6 00 00 00 00  01 00 00 00 00 00 00 00  |..*.............|
00000220  ff ff bf 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000230  d7 ff bf 00 00 00 00 00  54 a5 8f 32 e6 35 ef 11  |........T..2.5..|
00000240  b7 ac 08 00 27 c4 37 c5  02 00 00 00 00 00 00 00  |....'.7.........|
00000250  80 00 00 00 80 00 00 00  b5 4a 6b 97 00 00 00 00  |.........Jk.....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000410  b4 2e 44 61 e6 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..Da.5......'.7.|
00000420  00 08 00 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17feffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
17feffe10  04 00 00 00 75 66 73 6d  69 72 00 00 00 00 00 00  |....ufsmir......|
17feffe20  00 00 00 00 d0 bf 79 7e  a5 80 72 a2 02 00 00 00  |......y~..r.....|
17feffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe df 7f 01  |................|
17feffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
17feffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17feffe70  00 e0 7f 01 00 00 00 b5  62 b7 d6 f2 93 8a f4 56  |........b......V|
17feffe80  55 e0 25 2c ec c4 25 00  00 00 00 00 00 00 00 00  |U.%,..%.........|
17feffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe10  b4 2e 44 61 e6 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..Da.5......'.7.|
17fffbe20  00 08 00 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
17fffbe30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
17ffffe10  28 e5 09 b4 00 00 00 00  ff ff bf 00 00 00 00 00  |(...............|
17ffffe20  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
17ffffe30  d7 ff bf 00 00 00 00 00  54 a5 8f 32 e6 35 ef 11  |........T..2.5..|
17ffffe40  b7 ac 08 00 27 c4 37 c5  df ff bf 00 00 00 00 00  |....'.7.........|
17ffffe50  80 00 00 00 80 00 00 00  b5 4a 6b 97 00 00 00 00  |.........Jk.....|
17ffffe60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Test case 2:

Code:

=>      40  12582832  ada2  GPT  (6.0G)
        40      2008        - free -  (1.0M)
      2048   4194304     1  freebsd-ufs  (2.0G)
   4196352   4194304     2  freebsd-ufs  (2.0G)
   8390656   4190208     3  freebsd-ufs  (2.0G)
  12580864      2008        - free -  (1.0M)

=>      40  12582832  ada3  GPT  (6.0G)
        40      2008        - free -  (1.0M)
      2048   4194304     1  freebsd-ufs  (2.0G)
   4196352   4194304     2  freebsd-ufs  (2.0G)
   8390656   4190208     3  freebsd-ufs  (2.0G)
  12580864      2008        - free -  (1.0M)

hexdump(1) of ada2 after creating GPT table, showing primary and secondary GPT table

Code:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff bf 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  5b 51 74 e2 00 00 00 00  01 00 00 00 00 00 00 00  |[Qt.............|
00000220  ff ff bf 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000230  d7 ff bf 00 00 00 00 00  73 92 35 31 eb 35 ef 11  |........s.51.5..|
00000240  b7 ac 08 00 27 c4 37 c5  02 00 00 00 00 00 00 00  |....'.7.........|
00000250  80 00 00 00 80 00 00 00  ab 8c ea 1a 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000410  7b 36 59 45 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |{6YE.5......'.7.|
00000420  00 08 00 00 00 00 00 00  ff 07 40 00 00 00 00 00  |..........@.....|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000480  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000490  7f f8 78 46 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..xF.5......'.7.|
000004a0  00 08 40 00 00 00 00 00  ff 07 80 00 00 00 00 00  |..@.............|
000004b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000500  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000510  af 28 76 49 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |.(vI.5......'.7.|
00000520  00 08 80 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
00000530  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe10  7b 36 59 45 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |{6YE.5......'.7.|
17fffbe20  00 08 00 00 00 00 00 00  ff 07 40 00 00 00 00 00  |..........@.....|
17fffbe30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe80  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe90  7f f8 78 46 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..xF.5......'.7.|
17fffbea0  00 08 40 00 00 00 00 00  ff 07 80 00 00 00 00 00  |..@.............|
17fffbeb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbf00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbf10  af 28 76 49 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |.(vI.5......'.7.|
17fffbf20  00 08 80 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
17fffbf30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
17ffffe10  f7 6f 57 a0 00 00 00 00  ff ff bf 00 00 00 00 00  |.oW.............|
17ffffe20  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
17ffffe30  d7 ff bf 00 00 00 00 00  73 92 35 31 eb 35 ef 11  |........s.51.5..|
17ffffe40  b7 ac 08 00 27 c4 37 c5  df ff bf 00 00 00 00 00  |....'.7.........|
17ffffe50  80 00 00 00 80 00 00 00  ab 8c ea 1a 00 00 00 00  |................|
17ffffe60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

hexdump(1) of ada2 after creating ada2p[1,2,3]/ada3p[1,2,3] gmirror(8), label "ufsgm1|2|3", primary and secondary GPT table highlighted

Rich (BB code):

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff bf 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  5b 51 74 e2 00 00 00 00  01 00 00 00 00 00 00 00  |[Qt.............|
00000220  ff ff bf 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000230  d7 ff bf 00 00 00 00 00  73 92 35 31 eb 35 ef 11  |........s.51.5..|
00000240  b7 ac 08 00 27 c4 37 c5  02 00 00 00 00 00 00 00  |....'.7.........|
00000250  80 00 00 00 80 00 00 00  ab 8c ea 1a 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000410  7b 36 59 45 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |{6YE.5......'.7.|
00000420  00 08 00 00 00 00 00 00  ff 07 40 00 00 00 00 00  |..........@.....|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000480  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000490  7f f8 78 46 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..xF.5......'.7.|
000004a0  00 08 40 00 00 00 00 00  ff 07 80 00 00 00 00 00  |..@.............|
000004b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000500  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000510  af 28 76 49 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |.(vI.5......'.7.|
00000520  00 08 80 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
00000530  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
800ffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
800ffe10  04 00 00 00 75 66 73 67  6d 31 00 00 00 00 00 00  |....ufsgm1......|
800ffe20  00 00 00 00 ed 30 90 9f  ba e2 2d e3 02 00 00 00  |.....0....-.....|
800ffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe ff 7f 00  |................|
800ffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
800ffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
800ffe70  00 00 80 00 00 00 00 67  b2 cf 97 f6 81 78 66 64  |.......g.....xfd|
800ffe80  f9 76 83 64 ac 67 aa 00  00 00 00 00 00 00 00 00  |.v.d.g..........|
800ffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1000ffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
1000ffe10  04 00 00 00 75 66 73 67  6d 32 00 00 00 00 00 00  |....ufsgm2......|
1000ffe20  00 00 00 00 50 17 f7 a5  2c a7 b7 c0 02 00 00 00  |....P...,.......|
1000ffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe ff 7f 00  |................|
1000ffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
1000ffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1000ffe70  00 00 80 00 00 00 00 99  91 e0 10 22 b9 5d e9 24  |...........".].$|
1000ffe80  57 fe 9a f5 8c 7d b2 00  00 00 00 00 00 00 00 00  |W....}..........|
1000ffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17feffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
17feffe10  04 00 00 00 75 66 73 67  6d 33 00 00 00 00 00 00  |....ufsgm3......|
17feffe20  00 00 00 00 92 d9 7f 86  0e 6f 1d 9f 02 00 00 00  |.........o......|
17feffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe df 7f 00  |................|
17feffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
17feffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17feffe70  00 e0 7f 00 00 00 00 58  d0 fd 30 96 01 b3 2a 1c  |.......X..0...*.|
17feffe80  63 02 ae 48 8f 97 fa 00  00 00 00 00 00 00 00 00  |c..H............|
17feffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe10  7b 36 59 45 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |{6YE.5......'.7.|
17fffbe20  00 08 00 00 00 00 00 00  ff 07 40 00 00 00 00 00  |..........@.....|
17fffbe30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbe80  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbe90  7f f8 78 46 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |..xF.5......'.7.|
17fffbea0  00 08 40 00 00 00 00 00  ff 07 80 00 00 00 00 00  |..@.............|
17fffbeb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbf00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbf10  af 28 76 49 eb 35 ef 11  b7 ac 08 00 27 c4 37 c5  |.(vI.5......'.7.|
17fffbf20  00 08 80 00 00 00 00 00  ff f7 bf 00 00 00 00 00  |................|
17fffbf30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
17ffffe10  f7 6f 57 a0 00 00 00 00  ff ff bf 00 00 00 00 00  |.oW.............|
17ffffe20  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
17ffffe30  d7 ff bf 00 00 00 00 00  73 92 35 31 eb 35 ef 11  |........s.51.5..|
17ffffe40  b7 ac 08 00 27 c4 37 c5  df ff bf 00 00 00 00 00  |....'.7.........|
17ffffe50  80 00 00 00 80 00 00 00  ab 8c ea 1a 00 00 00 00  |................|
17ffffe60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

As the black highlighted data in both test cases shows, the secondary GPT table in the last sector of the disk is intact.

gmirror(8) is writing the metadata on the providers (partitions), not disks last sector, nicely visible on the second, multi-partition test case.

The GPT partitions were added with -a 1m alignment, the test outcome is the same with partitions added with no alignment

gpw928 · Saturday at 10:17 PM

Thank you T-Daemon for spending the effort to clear that up. Much appreciated.

I think that the issue is with using the whole disk (not a partition) as the provider.

mer · Saturday at 10:40 PM

T-Daemon curious about the actual sequence of steps. One of Michael Lucas books talks about this and the steps are roughly (going by memory because I'm too lazy to grab the book off the shelf):
gpart destroy -F ada0
gpart destroy -F ada1
gpart create -s gpt ada0
gpart create -s gpt ada1
At this point, there has been zero partitioning actually done
gmirror label mirror0 ada0
gmirror insert mirror0 ada1
Now we've actually created the mirror, then do the partitioning on the mirror
gpart add -t freebsd-boot -l boot -s 512K mirror/mirror0
and so on.

I think by doing that, creating the partitions on the mirror device it may avoid the issues

gpw928 · Sunday at 12:39 AM

mer said:
I think by doing that, creating the partitions on the mirror device it may avoid the issues

Your scenario above describes the situation where both GPT and gmirror(8) are being presented with a whole disk, and my understanding is that they both want the last sector of that disk for their own use. I think that the documentation is muddled because GPT was invented long after geom(8) and it's providers.

I don't perceive any problem when gmirror(8) is presented with partitions, as demonstrated by T-Daemon, above.

mer · Sunday at 8:51 AM

gpw928, thanks, that's why I asked what step he did.

T-Daemon · Sunday at 1:49 PM

gpw928 said:
Thank you T-Daemon for spending the effort to clear that up. Much appreciated.

You are most welcome.

gpw928 said:
I think that the issue is with using the whole disk (not a partition) as the provider.

In the whole disk, GPT table inside the mirror scenario is also no collision between the secondary GPT table and the gmirror(8) metadata.

I wasn't expecting a problem anyway. Think the gmirror(8) created mirror as an container. What happens inside doesn't spill over outside the container. Similar what happens in a geli(8) provider (partition or whole disk).

Test case 3:

gmirror(8) whole disk (no GPT partition table), label "ufsmir"

Code:

 # hexdump -C /dev/ada0
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
17ffffe10  04 00 00 00 75 66 73 6d  69 72 00 00 00 00 00 00  |....ufsmir......|
17ffffe20  00 00 00 00 44 24 22 19  07 8e d2 dc 02 00 00 00  |....D$".........|
17ffffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe ff 7f 01  |................|
17ffffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
17ffffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe70  00 00 80 01 00 00 00 1e  2b b4 a7 cf 72 73 e5 b3  |........+...rs..|
17ffffe80  4b b9 0e 18 9b 09 6d 00  00 00 00 00 00 00 00 00  |K.....m.........|
17ffffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
180000000

gmirror(8) whole disk, GPT table inside /dev/mirror/ufsmir created

Code:

# gpart show mirror/ufsmir
=>      40  12582832  mirror/ufsmir  GPT  (6.0G)
        40  12582832              1  freebsd-ufs  (6.0G)

Rich (BB code):

 # hexdump -C /dev/ada0
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 fe ff bf 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  57 3c 82 e7 00 00 00 00  01 00 00 00 00 00 00 00  |W<..............|
00000220  fe ff bf 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000230  d7 ff bf 00 00 00 00 00  e8 5e 6d 47 cf 36 ef 11  |.........^mG.6..|
00000240  86 aa 08 00 27 3a 03 f3  02 00 00 00 00 00 00 00  |....':..........|
00000250  80 00 00 00 80 00 00 00  2e 3f 1f 52 00 00 00 00  |.........?.R....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000410  e5 95 cb 50 cf 36 ef 11  86 aa 08 00 27 3a 03 f3  |...P.6......':..|
00000420  28 00 00 00 00 00 00 00  d7 ff bf 00 00 00 00 00  |(...............|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17fffbc00  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
17fffbc10  e5 95 cb 50 cf 36 ef 11  86 aa 08 00 27 3a 03 f3  |...P.6......':..|
17fffbc20  28 00 00 00 00 00 00 00  d7 ff bf 00 00 00 00 00  |(...............|
17fffbc30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffc00  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
17ffffc10  10 2f 28 6f 00 00 00 00  fe ff bf 00 00 00 00 00  |./(o............|
17ffffc20  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
17ffffc30  d7 ff bf 00 00 00 00 00  e8 5e 6d 47 cf 36 ef 11  |.........^mG.6..|
17ffffc40  86 aa 08 00 27 3a 03 f3  de ff bf 00 00 00 00 00  |....':..........|
17ffffc50  80 00 00 00 80 00 00 00  2e 3f 1f 52 00 00 00 00  |.........?.R....|
17ffffc60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe00  47 45 4f 4d 3a 3a 4d 49  52 52 4f 52 00 00 00 00  |GEOM::MIRROR....|
17ffffe10  04 00 00 00 75 66 73 6d  69 72 00 00 00 00 00 00  |....ufsmir......|
17ffffe20  00 00 00 00 44 24 22 19  07 8e d2 dc 02 00 00 00  |....D$".........|
17ffffe30  00 01 00 00 00 00 00 10  00 00 02 00 fe ff 7f 01  |................|
17ffffe40  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
17ffffe50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
17ffffe70  00 00 80 01 00 00 00 1e  2b b4 a7 cf 72 73 e5 b3  |........+...rs..|
17ffffe80  4b b9 0e 18 9b 09 6d 00  00 00 00 00 00 00 00 00  |K.....m.........|
17ffffe90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
180000000

The highlighted gmirror(8) metadata is in the last sector of the provider (disk in this case), the primary and secondary GPT partition table inside the mirror are all present and intact.

After all this testings, the question arises from where does the view come from, that GPT secondary partition table and gmirror(8) metadata don't get along.

Was it a myth all along, or is there a scenario which triggers the issue?

T-Daemon · Sunday at 1:52 PM

mer said:
T-Daemon curious about the actual sequence of steps.

The sequence of steps in test case 1 and 2 are standard gpart(8) create partition scheme, add partitions, then gmirror(8) label partitions.

mer said:
One of Michael Lucas books talks about this and the steps are roughly (going by memory because I'm too lazy to grab the book off the shelf):
[...]
gpart create -s gpt ada0
gpart create -s gpt ada1
At this point, there has been zero partitioning actually done
gmirror label mirror0 ada0
gmirror insert mirror0 ada1
Now we've actually created the mirror, then do the partitioning on the mirror
gpart add -t freebsd-boot -l boot -s 512K mirror/mirror0

You must be misremembering, it isn't logical. Why create a empty GPT table on the disks (ada0, ada1) and gmirror(8) the whole disks (ada0, ada1) afterwards?

mer said:
I think by doing that, creating the partitions on the mirror device it may avoid the issues

There is no problem with whole disks either, see "Test case 3" in posting # 11.

mer · Sunday at 2:06 PM

Now you got me curious, so I grabbed the book FreeBSD Mastery: Storage Essentials. Chapter 11: Complex Installation, first thing is "Installing to Gmirror" on pg 185. Pg 186-188 basically has the steps I laid out. Not sure why, I've never done the test (never cared to).
Speculation on my part:
simply creating the scheme doesn't try to do anything to the secondary location, since the actual partitions are created on the GEOM device, GEOM says "sector for secondary table is one less than physical size because I'm using last sector".
Effectively, your "gmirror the whole device then gpart the mirror".

One would have to ask Mr Lucas "why" on the create schema.

T-Daemon · Sunday at 2:19 PM

mer said:
One would have to ask Mr Lucas "why" on the create schema.

Not only the create schema, but also

mer said:
gmirror label mirror0 ada0
gmirror insert mirror0 ada1

Why not simply gmirror label mirror0 ada0 ada1

Erichans · Sunday at 8:38 PM

T-Daemon said:
After all this testings, the question arises from where does the view come from, that GPT secondary partition table and gmirror(8) metadata don't get along.

Info in this forum thread: EFI partition - mount_msdosfs: /dev/mirror/p1: Invalid argument.

gunion(8) and general GEOM info: Gunion(8): a new GEOM utility in the FreeBSD Kernel By: Marshall Kirk McKusick - BSDCAN 2023

gpw928 · 2024-07-01T02:50:43+0100

Thanks to everyone. This discussion has cleared up my long-standing dissonance regarding GEOM providers and GPT.

To summarise my understanding, for those who get here via google...

The rules of GPT say that "the backup GPT Header must be located in the last LBA of the device". GPT partitioning is thus not designed for recursive use (because there is only one last sector per device), so it's not natively suited to the stacking nature of GEOM.

When GPT is applied to a "disk" before GEOM, the GEOM providers available for deployment on the "disk" will be partitions, and the last sector of the "disk" is reserved for exclusive use of GPT, with no conflicts.

The are many good reasons to want to apply GEOM before GPT. The main one is to avoid head contention (one mirror to cover the entire disk, instead of one mirror per file system).

When GEOM is applied before GPT, GEOM places its metadata in the last sector of the provider, and reduces the advertised size of the provider by one sector. Subsequently, when a GPT partition table is installed on the GEOM, is installed onto the second last sector of the original provider. This behaviour violates the GPT rules -- the backup GPT Header must be located in the last LBA of the device (not the provider). But it allows GEOM to work in the FreeBSD context. If the disk is (re)located to a situation where GEOM is not operating, the backup GPT Header data structure will seem to be corrupt (because it's in the "wrong" place).

[There's a separate issue if you think you want to mirror an EFI partition. See post 4, above.]

fabric · 2024-07-01T16:37:43+0100

I found similar after experimenting a bit with the order...

When I gmirror the disks da0 and da1 (external data disks, of the same size) with 'gmirror label mirror-test da0 da1'

and check the disk sector size with diskinfo it reports the size as 1 sector less than the disk
i.e. 'diskinfo -v /dev/mirror/mirror-test' is one sector less than 'diskinfo -v /dev/da0'

Then create the GPT table on the mirror, 'gpart create -s gpt /dev/mirror/mirror-test'
the backup GPT can be see exactly one sector before the GEOM::MIRROR entry in the last sector; so they remain separate.

I also found, as others have above, mirroring smaller partitions seems fine - although haven't actually tried to use it in anger yet.

Thanks for all the discussion here - I think I understand it a little better now.

mer · 2024-07-01T18:21:43+0100

My speculation on the "gmirror of partitions":
GPT tables store the start end of the partitions, at start of device and end of device, nothing in the partition itself.
Why? Normally you after creating a partition you create a filesystem on it, say UFS. UFS stores metadata (superblocks) , probably not in the last sector of the partition, so when you gmirror partitions, the GEOM stuff is in the last sector of the partition and does not clash with partitioning or filesystem metadata.

I've not used gmirror for anything other than mirroring swap partitions: I'm a "use ZFS for everything on my systems", I do partitions for the ZFS bits and mirror them, so I have ZFS mirrored boot devices that are partitioned for swap, so I then use gmirror to create a GEOM mirror of the swap partitions.

Jose · 2024-07-01T20:57:29+0100

Well, I encountered the GPT error when I mirrored raw devices in Freebsd 11. I chose to mirror the devices, instead of partitions, because that way the mirroring happens at the block level, and things like UEFI partitions (which I don't use) or master boot records (which I do use) are also mirrored. I have the mirrored SSDs set up in the BIOS as boot devices 1 and 2, so in the event of a failure in the boot disk, the system will boot from the secondary device without requiring any intervention.

Phishfry · 2024-07-01T22:12:39+0100

5 Years later and i still have zero problems. Used on many machines.

Install UEFI FreeBSD on gmirror

I wanted to post instruction for installing FreeBSD on a GEOM_MIRROR -aka- gmirror(8). This is an advanced topic so I assume you are capable of determining that your two chosen disks are empty. UFS RAID1 on FreeBSD is enabled with the geom_mirror module. I am using ada0 and ada1 as examples...

forums.freebsd.org

I am glad I ignored the naysayers and just did it.
DutchDaemon had a RAID10 HowTo with gmirror & gstripe that inspired me to just try things.

I hate UEFI but I like the ease booting. Boot to UEFI OS and let it deal with the gmirror UUID instead of devices...

Phishfry · 2024-07-01T22:41:47+0100

Instead of worrying about sectors give the array the yank test.

Yank a drive and shut down the machine. Does it still boot after without intervention?

Does adding a new drive back work automatically? Can you boot off a new resynched member alone? Yes and Yes.
To me it has everything I need.

gpw928 · 2024-07-02T02:41:29+0100

Phishfry said:
5 Years later and i still have zero problems. Used on many machines.

That's a positive contribution for a plausible solution. Thank you. To re-state the caveats...

It only works with GPT disks because FreeBSD breaks the GPT rules regarding placement of the backup GPT Header. Thus the BIOS would not be able to use the backup GPT Header (unlikely, I admit).

[It works perfectly well with MBR disks, because there is no clash regarding metadata placement.]

It only works when an EFI partition is present if you can guarantee that the EFI shell (invoked from the BIOS) will never be used to (independently) modify anything in the EFI partition. A trap for novices, the forgetful, and where there's more than one admin.

[Some Linux systems, e.g. Ubuntu, now automate the solution by allowing you to: mount /efi and /efi2 and the Linux utilities that deal with boot code automatically keep them in sync.]