Other What happened to /dev/ada1s* ?

Esteemed Colleagues:

A couple weeks ago I began a thread on this forum ("Three Questions About Filesystems") in which I asked 3 questions. The 1st one I answered myself, the other 2 remain unanswered, and if you have a moment to check them out it would be appreciated. But that is not the reason why I write today.

I write today because, after re-installing FreeBSD, I am experiencing a new problem that I never experienced before. My computer has 2 hard drives, each one is partitioned, I have 11 different operating systems installed on various slices of the 2 disks, FreeBSD happens to be installed on /dev/ada0s5 (more precisely, root is on /dev/ada0s5a and swap is on /dev/ada0s5b) and now when I boot into the FreeBSD system, it appears to have no knowledge of the /dev/ada1 partition table (I wrote "appears to have", because there is more to tell, which you will see toward the end of this posting). Behold:

# ls /dev
acpi        ada1        cuau0.init    io        mixer1        random        ttyv4        ugen4.1
ada0        apm        cuau0.lock    kbd0        netmap        reroot        ttyv5        ugen5.1
ada0s1        apmctl        devctl        kbd1        nfslock        ses0        ttyv6        ugen6.1
ada0s10        atkbd0        devctl2        kbdmux0        null        sndstat        ttyv7        ugen7.1
ada0s11        audit        devstat        klog        nvidia0        stderr        ttyv8        ulpt0
ada0s2        auditpipe    diskid        kmem        nvidiactl    stdin        ttyv9        unlpt0
ada0s3        bpf        dsp0.0        led        pass0        stdout        ttyva        urandom
ada0s4        bpf0        dsp1.0        linux_lvm    pass1        sysmouse    ttyvb        usb
ada0s5        bpsm0        ext2fs        log        pass2        ttyu0        ufssuspend    usbctl
ada0s5a        cd0        fd        lpt0        pass3        ttyu0.init    ugen0.1        xpt0
ada0s5b        cd1        fido        lpt0.ctl    pass4        ttyu0.lock    ugen1.1        zero
ada0s6        console        full        mdctl        pci        ttyv0        ugen1.2        zfs
ada0s7        consolectl    fuse        mem        ppi0        ttyv1        ugen1.3
ada0s8        ctty        geom.ctl    midistat    psm0        ttyv2        ugen2.1
ada0s9        cuau0        hpet0        mixer0        pts        ttyv3        ugen3.1

The partition table unquestionably exists, however. First of all, there is the following error message that appears on the console at an early stage of booting, which I shall paste below (thank you, moused) surrounded by some context:

ada0: Serial Number WD-WCASY6811114
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3750640NS 3CNR> ATA-7 SATA 2.x device
ada1: Serial Number 5QD3PWEW
ada1: 300.000MB/s transfersuhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
 (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 715404MB (1465149168 512 byte sectors)
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1200003692 Hz quality 1000
Trying to mount root from ufs:/dev/ada0s5a [rw]...
uhub4: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
GEOM: ada1s7: invalid disklabel.
GEOM: diskid/DISK-5QD3PWEWs7: invalid disklabel.
Setting hostuuid: 44454c4c-4c00-1042-804d-c4c04f4d4e31.
Setting hostid: 0x79f7db45.
Starting file system checks:
/dev/ada0s5a: clean, 1851810 free (211378 frags, 205054 blocks, 3.5% fragmentation)
Mounting local filesystems:.
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/R /lib /usr/local/lib/gcc48 /usr/local/lib/gegl-0.2 /usr/local/lib/graphviz /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/opencollada /usr/local/lib/perl
5/5.20/mach/CORE /usr/local/lib/qt4 /usr/local/llvm37/lib
32-bit compatibility ldconfig path: /usr/lib32
Loading kernel modules:
GEOM_LINUX_LVM: disk pv1 already initialised in m5
Setting hostname: m5.
Feeding entropy: .
ugen1.2: <Hewlett-Packard> at usbus1
rl0: link state changed to DOWN
ugen1.3: <EPSON> at usbus1
rl0: link state changed to UP

In Line 20 of the above, there was a reference to ada1s7, specifically, a reference to an invalid disklabel (/dev/ada1s7 contains my OpenBSD system, which, like my FreeBSD and NetBSD systems, contains a BSD label at the beginning of the slice that defined the subpartitioning). Obviously, in order to complain about ada1s7, the ada1 partition table must be visible. And yet, the ls /dev output shows no evidence that it is.

Now, it would be facile for the reader to propose that the error encountered at ada1s7 caused FreeBSD to discard all knowledge of the ada1 partition table. However, when I boot from the installation DVD, I see the exact same error message (which I cannot paste here, because moused does not run when you boot into the Live DVD, but I assure you that it is the exact same error message), and yet, when I log in to the Live DVD and type ls /dev I see the entire /dev/ada1 partition table, including the subpartitioning of /dev/ada1s8. So there is something happening in my installed system that causes FreeBSD to lose knowledge of the /dev/ada1 partition table, which does not happen in the Live DVD.

There is further evidence that the ada1 partition table is visible at an early stage of booting. I have a Linux volume group, comprising two physical volumes, one of which is a slice of /dev/ada0 and one of which is a slice of /dev/ada1. I have geom_linux_lvm loaded, which you can verify from the following output:

# kldstat
Id Refs Address            Size     Name
 1   43 0xffffffff80200000 1fa7c38  kernel
 2    1 0xffffffff821a9000 e124c8   nvidia.ko
 3    2 0xffffffff82fbc000 9b748    linux.ko
 4    4 0xffffffff83058000 de28     linux_common.ko
 5    1 0xffffffff83066000 1a7c8    fuse.ko
 6    1 0xffffffff83221000 587b     fdescfs.ko
 7    1 0xffffffff83227000 a9f1     linprocfs.ko
 8    1 0xffffffff83232000 52df     geom_linux_lvm.ko
 9    1 0xffffffff83238000 249d     ulpt.ko
10    1 0xffffffff8323b000 389f4    linux64.ko
11    1 0xffffffff83274000 1fe5a3   zfs.ko
12    1 0xffffffff83473000 811f     opensolaris.ko
13    1 0xffffffff8347c000 155b2    ext2fs.ko

and every one of my logical volumes is visible (except for the mirrored volumes, however each one of the mirror images is visible):

# ls -l /dev/linux_lvm
total 0
crw-r-----  1 root  operator  0x9e Jan 24 14:34 m5-SLES-old
crw-r-----  1 root  operator  0x9a Jan 24 14:34 m5-SLES_mimage_0
crw-r-----  1 root  operator  0x7e Jan 24 14:34 m5-SLES_mimage_1
crw-r-----  1 root  operator  0x85 Jan 24 14:34 m5-SLES_mlog
crw-r-----  1 root  operator  0x91 Jan 24 14:34 m5-old-suse
crw-r-----  1 root  operator  0x9c Jan 24 14:34 m5-rhel_mimage_0
crw-r-----  1 root  operator  0x90 Jan 24 14:34 m5-rhel_mimage_1
crw-r-----  1 root  operator  0x9d Jan 24 14:34 m5-rhel_mlog
crw-r-----  1 root  operator  0x9f Jan 24 14:34 m5-scientific_linux
crw-r-----  1 root  operator  0xa9 Jan 24 14:34 m5-swapa
crw-r-----  1 root  operator  0x99 Jan 24 14:34 m5-swapb
crw-r-----  1 root  operator  0x9b Jan 24 14:34 m5-ubuntu_mimage_0
crw-r-----  1 root  operator  0x8e Jan 24 14:34 m5-ubuntu_mimage_1
crw-r-----  1 root  operator  0x8f Jan 24 14:34 m5-ubuntu_mlog

But each of the mirrored volumes comprises two mirrored images, the first of which is entirely resident on the physical volume that resides on ada0 (specifically, /dev/ada0s11), and the second of which is entirely resident on the physical volume that resides on ada1 (specifically, /dev/ada1s5). Moreover, the swapb volume is entirely resident on the physical volume that resides on ada1. Now, it seems to me that if FreeBSD can find logical volumes that reside entirely within a slice of ada1, it must have had access to the ada1 partition table that enabled it to find that slice.

And now for the most confusing part, which I alluded to earlier: /dev/diskid contains entries for the ada1 slices -- i.e., the ada1s* entries that ought to have been directly underneath /dev -- and only the ada1 slices, it contains no entries for the ada0 slices. Behold:

# ls -las /dev/diskid/
total 1
1 dr-xr-xr-x   2 root  wheel      512 Jan 24 14:34 .
1 dr-xr-xr-x  14 root  wheel      512 Jan 24 14:34 ..
0 crw-r-----   1 root  operator  0x96 Jan 24 14:34 DISK-5QD3PWEW
0 crw-r-----   1 root  operator  0xa5 Jan 24 14:34 DISK-5QD3PWEWs1
0 crw-r-----   1 root  operator  0xa6 Jan 24 14:34 DISK-5QD3PWEWs2
0 crw-r-----   1 root  operator  0xa7 Jan 24 14:34 DISK-5QD3PWEWs3
0 crw-r-----   1 root  operator  0xa8 Jan 24 14:34 DISK-5QD3PWEWs4
0 crw-r-----   1 root  operator  0xaf Jan 24 14:34 DISK-5QD3PWEWs5
0 crw-r-----   1 root  operator  0xb0 Jan 24 14:34 DISK-5QD3PWEWs6
0 crw-r-----   1 root  operator  0xb1 Jan 24 14:34 DISK-5QD3PWEWs7
0 crw-r-----   1 root  operator  0xb2 Jan 24 14:34 DISK-5QD3PWEWs8
0 crw-r-----   1 root  operator  0xb5 Jan 24 14:34 DISK-5QD3PWEWs8k
0 crw-r-----   1 root  operator  0xb6 Jan 24 14:34 DISK-5QD3PWEWs8l
0 crw-r-----   1 root  operator  0xb3 Jan 24 14:34 DISK-5QD3PWEWs9

Note that there are even entries for the NetBSD subpartitioning on slice 8, although not for the OpenBSD subpartitioning on slice 7, presumably because of the ada1s7 error that the boot procedure complained about (and which nevertheless did not prevent the Live DVD from creating /dev/ada1s*).

Moreover, the gpart command works correctly on these items (except for the failure to find the OpenBSD subpartitions). Behold:

# gpart show diskid/DISK-5QD3PWEW
=>        63  1465149105  diskid/DISK-5QD3PWEW  MBR  (699G)
          63        1985                        - free -  (993K)
        2048   102460522                     1  ntfs  (49G)
   102462570    73738350                     2  !191  [active]  (35G)
   176200920   503316480                     3  linux-data  (240G)
   679517400   785631768                     4  ebr  (375G)

# gpart show diskid/DISK-5QD3PWEWs4
=>        0  785631768  diskid/DISK-5QD3PWEWs4  EBR  (375G)
          0  146286640                       1  linux-lvm  (70G)
  146286640   41945088                 2322011  !48  (20G)
  188231728   67110912                 2987806  !166  (32G)
  255342640        760                          - free -  (380K)
  255343400   75499520                 4053070  !169  (36G)
  330842920  454788848                 5251475  linux-data  (217G)

# gpart show diskid/DISK-5QD3PWEWs8
=>       0  75497472  diskid/DISK-5QD3PWEWs8  BSD  (36G)
         0  61440320                      12  freebsd-ufs  (29G)
  61440320  14057152                      11  freebsd-swap  (6.7G)

[root@m5 /opt/assp]# gpart show ada1
gpart: No such geom: ada1.

My guess is that geom_linux_lvm has something to do with this, although I do not know how. I have observed that, in the Live DVD, if I kldload geom_linux_lvm the /dev/linux_lvm directory springs into existence with all its contents, and then if I mount /dev/ada0s5a /mnt, the /dev/linux_lvm directory immediately disappears. There is a similar phenomenon in the installed system -- which did not formerly occur, but now it does -- such that I do not have post-startup access to /dev/linux_lvm if I put "geom_linux_lvm_load=YES" into /boot/loader.conf (if the startup messages are to be believed, the directory is created and then later its contents are removed); I only have post-startup access to /dev/linux_lvm if I employ the technique of saying "kld_list=geom_linux_lvm" in /etc/rc.conf, I am guessing because the former technique loads the module before the root filesystem is mounted whereas the latter technique mounts the root filesystem first and then loads the module. On the other hand, this could be an entirely unrelated phenomenon, albeit an interesting one in its own right.

In case you were thinking of suggesting it, service devd restart does not create the ada1s* entries in /dev. So, what can I do to restore the /dev/ada1s* entries to which I am accustomed, or am I condemned, for mysterious reasons, to henceforward accessing them underneath /dev/diskid? As always, thank you in advance for any and all replies.
I'd try adding the following to /boot/loader.conf first to disable those diskid entries (then reboot):

There does seem to be a weird behaviour in FreeBSD where some disk device nodes will disappear if the same device is used via another node. For example you may find the linux LVM geom provider is opening the /dev/diskid/* nodes, causing the /dev/ada1* devices to disappear. I'm sure I've seen references to this effect in documentation somewhere but personally I find the whole thing a bit of a mess. I tend to stick to GPT partitions these days (ada1, ada1p1, ada1p2, etc), use GPT labels and disable everything else.
There does seem to be a weird behaviour in FreeBSD where some disk device nodes will disappear if the same device is used via another node.

This is called GEOM "withering." Jay F. Shachter, you repeatedly mention FreeBSD being unaware of the partition table, but as you pointed out, the system sees the disk and partitions just fine. It's just not providing device nodes you expect to see in /dev. GEOM (the FreeBSD storage framework) often offers multiple nodes through which to access storage devices, but only allows one of those nodes to be used at a time. All other nodes are deleted, and from my experience I get the impression that when multiple device nodes might be chosen from, the system prioritizes which one to present to the user, with plain-old drive letters being the lowest priority. So just because ada1 doesn't exist doesn't mean the system can't see the disk. It's just presenting the disk to you using a higher-priority device node.

It seems many users are confused by this when they first come across it, but there is a logic to it---the worst way to present disks to users is as a series of arbitrary alphanumeric labels. I do the same that usdmatt does, but obviously if you're using lots of different operating systems with different storage schemes you might be stuck with MBR tables.