bhyve Stability in nvme/nda device numbering?

[Chris I.]

Just encountered an issue while migrating a vm from a 14.4 host to a 15.1 host. The VM has failed to complete boot process because of a fstab device numbering conflict. The VM has three nvme devices configured, and it appears that the 2nd and 3rd nda devices were enumerated out of order. (At least I think that is what I saw.)

Has nda device enumeration always been indeterminate ? or is this maybe a new issue with FreeBSD 15 hosts or churchers/vm-bhyve ?

We resolved the problem by using GPT partition label and updating fstab to avoid device numbers. I've used this method for other unpredictable devices for some time, but I don't recall encountering it yet with nvme.

But of course, the issue didn't repeat on the next boot, so there at least a small chance I imagined the whole thing.

Host Machine:
FreeBSD 15.1-RELEASE-p1
vm-bhyve-1.7.3

vm-bhyve devices log excerpt on recent/working boot:

-s 0:4:0,nvme,/...disk0.img -s 0:4:1,nvme,/...disk1.img -s 0:4:2,nvme,/...disk2.img

vm-bhyve config excerpt:

disk0_type="nvme"
disk0_name="disk0.img"

disk1_type="nvme"
disk1_name="disk1.img"

disk2_type="nvme"
disk2_name="disk2.img"

Guest VM:
FreeBSD 14.4-RELEASE-p1

$ nvmecontrol devlist -h

 nvme0: bhyve-NVMe
    nvme0ns1 (20GB)
 nvme1: bhyve-NVMe
    nvme1ns1 (210GB)
 nvme2: bhyve-NVMe
    nvme2ns1 (150GB)

$ pciconf -l

hostb0@pci0:0:0:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1275 device=0x1275 subvendor=0x0000 subdevice=0x0000
nvme0@pci0:0:4:0:    class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
nvme1@pci0:0:4:1:    class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
nvme2@pci0:0:4:2:    class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
virtio_pci0@pci0:0:5:0:    class=0x020000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1000 subvendor=0x1af4 subdevice=0x0001
isab0@pci0:0:31:0:    class=0x060100 rev=0x00 hdr=0x00 vendor=0x8086 device=0x7000 subvendor=0x0000 subdevice=0x0000

$ gpart show
=>      34  41942966  nda0  GPT  (20G)
        34       122     1  freebsd-boot  (61K)
       156     66584     2  efi  (33M)
     66740   2097152     3  freebsd-swap  (1G)
   2163892  39779108     4  freebsd-ufs  (19G)

=>       40  440401840  nda1  GPT  (210G)
         40  440401840     1  freebsd-ufs  (210G)

=>       40  314572720  nda2  GPT  (150G)
         40  314572720     1  freebsd-ufs  (150G)

 

[Maturin]

We resolved the problem by using GPT partition label and updating fstab to avoid device numbers.

That's the way to do it, and very recommended to be done in general always on any machine with more than one drive.

The device numbers are given very early at booting in the order of the devices are detected. The fastest drive detected first becomes 0, the second 1, etc.
If for whatever reasons something is changed - drives added, replaced, removed, connection ports switched, or the access time of a drive change, which I never seen happen with real physical drives by now, but I can imagine such could happen with virtual drives being files in a filesystem - the order of detection, so the numbering, and so the device names can change.
Since FreeBSD only boots correctly to the full if every entry in the fstab fits exactly the according drive, a change of a single drive's name will make the fstab not being fully correct anymore, and so the sysetm will not boot anymore properly.

With labels this all can be avoided, since the label is linked to a specific physical drive, no matter which device number it's given, or if that changes.

One may also check out additional options drives can be given in fstab for being ignored if not detected, or not being mounted automatically by default, which of course makes only sense for optional filesystems, but is pointless to the drive that contains / or others system's filesystems the system must have.