/dev/mirror missing after upgrade

P

ppouliquen

I just did a upgrade from 13.3 to 14.2. After rebooting, my gmirror wouldn't mount, and I discovered that the entire /dev/mirror directory was missing. It seems that I could just re-issue the gmirror label command that I used originally to create the mirror, but it's not clear from the gmirror man page or the FreeBSD handbook if this will trash the existing mirror contents or not.

On this system, there are three SSDs with the following partitions:
  • /dev/nvd2p1 (EFI), /dev/nvd2p2 (ROOT), /dev/nvd2p3 (SWAP)
  • /dev/nvd1p1 (gmirror gm0, priority 1)
  • /dev/nvd0p1 (gmirror gm0, priority 0)
The mirror was originally created with the command:

gmirror label -v gm0 /dev/nvd0p1 /dev/nvd1p1


I checked that the gmirror metadata is still present on the two providers using the command gmirror dump /dev/nvd0p1 /dev/nvd1p1.
 
Maybe it's a hint to know that since 14 the default names the system gives nvme-drives changed from former nvd0 to now nda0.
I had a similar issue once when upgrading from 13.2 to 14.0. In my case it was quickly solved by correcting /etc/fstab,
or better use labels in the first place.
 
The sysctl kern.geom.mirror.debug described in gmirror(8) might be useful. It can also be set from the loader. The extra logging that it enables will appear in dmesg.

The directory /dev/mirror automatically appears whenever there are gmirror devices to show in it.
 
It shouldn't be a problem with device names. Old nvd device names are symbolically linked to new nda device names:
Code: 
% ls -l /dev/nvd0*
lrwxr-xr-x  1 root wheel  4 May 26 10:14 /dev/nvd0 -> nda0
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p1 -> nda0p1
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p2 -> nda0p2
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p3 -> nda0p3

ppouliquen, what does gmirror dump /dev/nvd0p1 /dev/nvd1p1 exactly show? Please show the whole dump.
 
ppouliquen said:
I just did a upgrade from 13.3 to 14.2.
Click to expand...
We should also check freebsd-version -kru if kernel and userland are the same version (in particular kernel module and userland /sbin/gmirror are of interest ). Also, check /var/log/messages if there are gmirror related
 
T-Daemon said:
It shouldn't be a problem with device names. Old nvd device names are symbolically linked to new nda device names:
Code: 
% ls -l /dev/nvd0*
lrwxr-xr-x  1 root wheel  4 May 26 10:14 /dev/nvd0 -> nda0
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p1 -> nda0p1
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p2 -> nda0p2
lrwxr-xr-x  1 root wheel  6 May 26 10:14 /dev/nvd0p3 -> nda0p3

ppouliquen, what does gmirror dump /dev/nvd0p1 /dev/nvd1p1 exactly show? Please show the whole dump.
Click to expand...
Code: 
# gmirror dump /dev/nvd0p1 /dev/nvd1p1
Metadata on /dev/nvd0p1:
     magic: GEOM::MIRROR
   version: 4
      name: gm0
       mid: 4263148793
       did: 2053231551
       all: 2
     genid: 0
    syncid: 1
  priority: 0
     slice: 4096
   balance: load
 mediasize: 1000204844544
sectorsize: 512
syncoffset: 0
    mflags: NONE
    dflags:
hcprovider: nvd0p1
  provsize: 1000204845056
  MD5 hash: 2a0154183f2476bdd2dd728d477a29df

Metadata on /dev/nvd1p1:
     magic: GEOM::MIRROR
   version: 4
      name: gm0
       mid: 4263148793
       did: 2412283868
       all: 2
     genid: 0
    syncid: 1
  priority: 1
     slice: 4096
   balance: load
 mediasize: 1000204844544
sectorsize: 512
syncoffset: 0
    mflags: NONE
    dflags:
hcprovider: nvd1p1
  provsize: 1000204845056
  MD5 hash: 27865cffede3fc2b0f029f1af76b9305
 
T-Daemon said:
We should also check freebsd-version -kru if kernel and userland are the same version (in particular kernel module and userland /sbin/gmirror are of interest ). Also, check /var/log/messages if there are gmirror related
Click to expand...
There is a discrepancy there:
Code: 
# freebsd-version -kru
14.2-RELEASE-p1
14.2-RELEASE-p1
14.2-RELEASE-p3

Nothing interesting in /var/log/messages. grep -i -e mirror -e geom /var/log/messages returns nothing. I also ran:

Code: 
# dmesg | grep -i -e geom -e mirror -e "RELEASE"
FreeBSD 13.3-RELEASE-p7 GENERIC amd64
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
FreeBSD 14.2-RELEASE-p1 GENERIC amd64
FreeBSD 14.2-RELEASE-p1 GENERIC amd64
 
ppouliquen said:
There is a discrepancy there:
Code: 
# freebsd-version -kru
14.2-RELEASE-p1
14.2-RELEASE-p1
14.2-RELEASE-p3
Click to expand...
Don't worry, this is quite alright.

The -p1's are showing the patch level of the running and installed kernel. The latest two upgrades (p2,p3) didn't involved the kernel, only userland.

I wanted to see if perhaps a incomplete system upgrade happened, leaving the kernel or userland on 13.3. But that's not the case.

I did some testing, but couldn't reproduce the issue.

In a bhyve(8) test VM (disk type "nvme", 3 disks, 1 for OS, 2 reserved for data gmirror), installed 13.3-RELEASE-p8, created a gmirror from two nvd*p1 devices, upgraded (from source) to 14.2. The gmirror is up and running on 14.2 without problems (switched to nda*p1 device names automatically).

Maybe ask on mailing lists, perhaps someone of the developers there has an idea what could cause the failure, I don't know how to debug the problem further.
 
I have some additional clues now: I tried the live CD on the FreeBSD-14.2-RELEASE-amd64-memstick.img, and after issuing gmirror load it also did not see the mirror. However, when I tried the Live CD on my old FreeBSD-13.0-RELEASE USB stick (from my drawer), and after issuing gmirror load it did see the mirror!

T-Daemon said:
I did some testing, but couldn't reproduce the issue.

In a bhyve(8) test VM (disk type "nvme", 3 disks, 1 for OS, 2 reserved for data gmirror), installed 13.3-RELEASE-p8, created a gmirror from two nvd*p1 devices, upgraded (from source) to 14.2. The gmirror is up and running on 14.2 without problems (switched to nda*p1 device names automatically).

Maybe ask on mailing lists, perhaps someone of the developers there has an idea what could cause the failure, I don't know how to debug the problem further.
Click to expand...

I appreciate the effort -- trying an install/update is more than I could ask for. If you still have the VM, could you run a gmirror dump?
 
T-Daemon said:
In a bhyve(8) test VM (disk type "nvme", 3 disks, 1 for OS, 2 reserved for data gmirror), installed 13.3-RELEASE-p8, created a gmirror from two nvd*p1 devices, upgraded (from source) to 14.2.
Click to expand...
I just noticed that you wrote from source -- do you think that this is a significant difference from using freebsd-update? What patch level did you end up on?
 
ppouliquen said:
I have some additional clues now: I tried the live CD on the FreeBSD-14.2-RELEASE-amd64-memstick.img, and after issuing gmirror load it also did not see the mirror. However, when I tried the Live CD on my old FreeBSD-13.0-RELEASE USB stick (from my drawer), and after issuing gmirror load it did see the mirror!
Click to expand...
That's unusual. Utilities and kernel modules, in general, are backwards compatible.

Try this: Disable gmirror kernel module (comment geom_mirror_load in loader.conf), reboot system, mount the 13.0 USB stick on the 14.2 system (it should be daXs2a, replace X with actual device number), kldload /mnt/boot/kernel/geom_mirror.ko, check gmirror status.

Second test: Get a another 14.2 geom_mirror.ko , load it. I've attached one to this post. If you are reluctant on using system components from unknown persons, get a kernel.txz distribution file from 14.2-memstick USB (/usr/freebsd-dist/kernel.txz) or download [1], extract module, kldload(8) by relative or full path :
Code: 
% tar xfv kernel.txz --strip-component 3 boot/kernel/geom_mirror.ko

ppouliquen said:
I appreciate the effort -- trying an install/update is more than I could ask for. If you still have the VM, could you run a gmirror dump?
Click to expand...
No problem. If I find a topic interesting, and time permits, I do, even lengthy testings, out of curiosity.

Not much helpful information in the dump I'm afraid. In my 14.2 systems dump "hcprovider" is empty, though.
Code: 
Metadata on nvd1p1:
     magic: GEOM::MIRROR
   version: 4
      name: GMIR0
       mid: 213110813
       did: 3426577212
       all: 2
     genid: 0
    syncid: 1
  priority: 0
     slice: 4096
   balance: load
 mediasize: 268433358336
sectorsize: 512
syncoffset: 0
    mflags: NONE
    dflags:
hcprovider:
  provsize: 268433358848
  MD5 hash: a9c3069974623927fec3d7417a3ea886

Metadata on nvd2p1:
     magic: GEOM::MIRROR
   version: 4
      name: GMIR0
       mid: 213110813
       did: 1533387462
       all: 2
     genid: 0
    syncid: 1
  priority: 1
     slice: 4096
   balance: load
 mediasize: 268433358336
sectorsize: 512
syncoffset: 0
    mflags: NONE
    dflags:
hcprovider:
  provsize: 268433358848
  MD5 hash: 522cb2e207c14ed4e7dcc32d45ba25d0

See if there are hints in the debug information JordanG and covacat suggested. While debug enable, you can boot the system without the kernel module enable, kldload modules after from different branches, check debug info then.

ppouliquen said:
I just noticed that you wrote from source -- do you think that this is a significant difference from using freebsd-update? What patch level did you end up on?
Click to expand...
There is no difference at all updating/upgrading the system from source or freebsd-update(8).

The patch level is basically the same as your system. Don't mind the kernel -p3 patch level. Although unnecessary, I did a kernel build/install. This elevated the kernel to -p3 but no patches were applied actually.
Code: 
% freebsd-version -kru
14.2-RELEASE-p3
14.2-RELEASE-p3
14.2-RELEASE-p3


[1] https://download.freebsd.org/releases/amd64/14.2-RELEASE/kernel.txz
 

Attachments

covacat said:
put kern.geom.mirror.debug=99 in loader.conf and reboot
Click to expand...
JordanG said:
The sysctl kern.geom.mirror.debug described in gmirror(8) might be useful. It can also be set from the loader. The extra logging that it enables will appear in dmesg.
Click to expand...
I placed the kern.geom.mirror.debug=99 in the loader.conf and commented out the geom_mirror_load="YES" so that I could try different kernel modules. (Sorry JordanG, I somehow missed your original reply.) With the 14.2 kernel module, I get:

Code: 
GEOM_MIRROR[2]: Tasting diskid/DISK-070A9705EB673D80p2.
GEOM_MIRROR[2]: Tasting diskid/DISK-070A9705EB673D80p1.
GEOM_MIRROR[2]: Tasting iso9660/13_0_RELEASE_AMD64_DVDp2.
GEOM_MIRROR[2]: Tasting iso9660/13_0_RELEASE_AMD64_DVDp1.
GEOM_MIRROR[2]: Tasting da4p2.
GEOM_MIRROR[2]: Tasting da4p1.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W506310Rp1.
GEOM_MIRROR[2]: Tasting nda2p3.
GEOM_MIRROR[2]: Tasting nda2p2.
GEOM_MIRROR[2]: Tasting nda2p1.
GEOM_MIRROR[2]: Tasting nda1p1.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W508224Bp1.
GEOM_MIRROR[2]: Tasting nda0p1.
GEOM_MIRROR[2]: Tasting msdosfs/EFISYS.
GEOM_MIRROR[2]: Tasting gptid/4b2d443d-9908-11eb-bf4c-002590ec5bf2.
GEOM_MIRROR[2]: Tasting gptid/4b2d4434-9908-11eb-bf4c-002590ec5bf2.
GEOM_MIRROR[2]: Tasting diskid/DISK-070A9705EB673D80.
GEOM_MIRROR[2]: Tasting iso9660/13_0_RELEASE_AMD64_DVD.
GEOM_MIRROR[2]: Tasting gptid/2204e969-7015-11ee-a4d2-d05099838bc5.
GEOM_MIRROR[2]: Tasting gpt/MIR1.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W506310R.
GEOM_MIRROR[2]: Tasting ufsid/6534546dcfb9367c.
GEOM_MIRROR[2]: Tasting gptid/2d176080-7015-11ee-a4d2-d05099838bc5.
GEOM_MIRROR[2]: Tasting gpt/MIR0.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W508224B.
GEOM_MIRROR[2]: Tasting da4.
GEOM_MIRROR[2]: Tasting da3.
GEOM_MIRROR[1]: Cannot read metadata from da3 (error=6).
GEOM_MIRROR[2]: Tasting da2.
GEOM_MIRROR[1]: Cannot read metadata from da2 (error=6).
GEOM_MIRROR[2]: Tasting da1.
GEOM_MIRROR[1]: Cannot read metadata from da1 (error=6).
GEOM_MIRROR[2]: Tasting da0.
GEOM_MIRROR[1]: Cannot read metadata from da0 (error=6).
GEOM_MIRROR[2]: Tasting nda2.
GEOM_MIRROR[2]: Tasting nda1.
GEOM_MIRROR[2]: Tasting nda0.

Note: drives da0, da1, da2, da3 are removable drive bays with no drives installed. da4 is the FreeBSD-13.0 install USB. nda2p1, nda2p2, nda2p3 are the FreeBSD-14.2 partitions (EFI, ROOT, SWAP). nda0p1 and nda1p1 are the mirror (although they were created with the old drive names of nvd0p1 and nvd1p1). MIR0 and MIR1 are the labels I used for the mirror partitions. I did not track down the gptid/* or the diskid/*.

T-Daemon said:
Try this: Disable gmirror kernel module (comment geom_mirror_load in loader.conf), reboot system, mount the 13.0 USB stick on the 14.2 system (it should be daXs2a, replace X with actual device number), kldload /mnt/boot/kernel/geom_mirror.ko, check gmirror status.
Click to expand...
I could not mount the 13.0 USB stick, it appears that it may be corrupt now (although I can still boot to the Live CD -- this may be another clue, so let me know if I should pursue it). Instead, I copied the 13.3 geom_mirror.ko from another working 13.3 system. I rebooted the 14.2 system and then:

Code: 
# kldload ./geom_mirror_13.3.ko
KLD geom_mirror_13.3.ko: depends on kernel - not available or version mismatch
kldload: can't load ./geom_mirror_13.3.ko: module already loaded or in kernel

Note: the first message is from dmesg -- I was running the command from the console.
T-Daemon said:
Second test: Get a another 14.2 geom_mirror.ko , load it. I've attached one to this post.
Click to expand...

I tried the version you attached:

Code: 
# kldload ./geom_mirror-142.ko.txt
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W506310Rp1.
GEOM_MIRROR[2]: Tasting nda2p3.
GEOM_MIRROR[2]: Tasting nda2p2.
GEOM_MIRROR[2]: Tasting nda2p1.
GEOM_MIRROR[2]: Tasting nda1p1.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W508224Bp1.
GEOM_MIRROR[2]: Tasting nda0p1.
GEOM_MIRROR[2]: Tasting gptid/2204e969-7015-11ee-a4d2-d05099838bc5.
GEOM_MIRROR[2]: Tasting gpt/MIR1.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W506310R.
GEOM_MIRROR[2]: Tasting ufsid/6534546dcfb9367c.
GEOM_MIRROR[2]: Tasting gptid/2d176080-7015-11ee-a4d2-d05099838bc5.
GEOM_MIRROR[2]: Tasting gpt/MIR0.
GEOM_MIRROR[2]: Tasting diskid/DISK-S5P2NU0W508224B.
GEOM_MIRROR[2]: Tasting da3.
GEOM_MIRROR[1]: Cannot read metadata from da3 (error=6).
GEOM_MIRROR[2]: Tasting da2.
GEOM_MIRROR[1]: Cannot read metadata from da2 (error=6).
GEOM_MIRROR[2]: Tasting da1.
GEOM_MIRROR[1]: Cannot read metadata from da1 (error=6).
GEOM_MIRROR[2]: Tasting da0.
GEOM_MIRROR[1]: Cannot read metadata from da0 (error=6).
GEOM_MIRROR[2]: Tasting nda2.
GEOM_MIRROR[2]: Tasting nda1.
GEOM_MIRROR[2]: Tasting nda0.

Note: I removed da4 for this test.

I wanted to see what the debug information was supposed to look like, but I ran into some trouble. The 13.0 install USB Live CD doesn't recognize the kern.geom.mirror.debug OID. It seems that the OID isn't available until the geom_mirror.ko is loaded, and then it's too late to turn debugging on. So I installed 13.0 on a compact flash (one of the removable drive bays is CF). Then I was able to set the debug OID in loader.conf, and capture the dmesg output when loading the kernel module. Setting it to 99 was probably a bit high -- the output is almost 700 lines, so I am attaching it instead of in-lining it.
 

Attachments

Maturin said:
Maybe it's a hint to know that since 14 the default names the system gives nvme-drives changed from former nvd0 to now nda0.
I had a similar issue once when upgrading from 13.2 to 14.0. In my case it was quickly solved by correcting /etc/fstab,
or better use labels in the first place.
Click to expand...
T-Daemon said:
Not much helpful information in the dump I'm afraid. In my 14.2 systems dump "hcprovider" is empty, though.
Click to expand...
Well, it looks like my problem is solved. In FreeBSD 13.0, I did:
Code: 
gmirror configure -d gm0

This removed the "hcprovider" entry from the metadata. Then, rebooting in FreeBSD 14.2, gmirror load worked!

I'm guessing that gmirror didn't like that the device names (nda0p1, nda1p1) didn't match the hcprovider entry. I'm also guessing that gmirror won't probe the nvd0p1and nvd1p1 entries because they are symbolic links. (I wonder if I could have fixed it by making the nvd entries hard links or using mknod to create copies of the nda devices.)

Thanks everyone for your input...
 
You must log in or register to reply here.
Back
Top