Just encountered an issue while migrating a vm from a 14.4 host to a 15.1 host. The VM has failed to complete boot process because of a fstab device numbering conflict. The VM has three nvme devices configured, and it appears that the 2nd and 3rd nda devices were enumerated out of order. (At least I think that is what I saw.)
Has nda device enumeration always been indeterminate ? or is this maybe a new issue with FreeBSD 15 hosts or churchers/vm-bhyve ?
We resolved the problem by using GPT partition label and updating fstab to avoid device numbers. I've used this method for other unpredictable devices for some time, but I don't recall encountering it yet with nvme.
But of course, the issue didn't repeat on the next boot, so there at least a small chance I imagined the whole thing.
Host Machine:
FreeBSD 15.1-RELEASE-p1
vm-bhyve-1.7.3
vm-bhyve devices log excerpt on recent/working boot:
vm-bhyve config excerpt:
Guest VM:
FreeBSD 14.4-RELEASE-p1
Has nda device enumeration always been indeterminate ? or is this maybe a new issue with FreeBSD 15 hosts or churchers/vm-bhyve ?
We resolved the problem by using GPT partition label and updating fstab to avoid device numbers. I've used this method for other unpredictable devices for some time, but I don't recall encountering it yet with nvme.
But of course, the issue didn't repeat on the next boot, so there at least a small chance I imagined the whole thing.
Host Machine:
FreeBSD 15.1-RELEASE-p1
vm-bhyve-1.7.3
vm-bhyve devices log excerpt on recent/working boot:
Code:
-s 0:4:0,nvme,/...disk0.img -s 0:4:1,nvme,/...disk1.img -s 0:4:2,nvme,/...disk2.imgvm-bhyve config excerpt:
Code:
disk0_type="nvme"
disk0_name="disk0.img"
disk1_type="nvme"
disk1_name="disk1.img"
disk2_type="nvme"
disk2_name="disk2.img"Guest VM:
FreeBSD 14.4-RELEASE-p1
Code:
$ nvmecontrol devlist -h
nvme0: bhyve-NVMe
nvme0ns1 (20GB)
nvme1: bhyve-NVMe
nvme1ns1 (210GB)
nvme2: bhyve-NVMe
nvme2ns1 (150GB)
$ pciconf -l
hostb0@pci0:0:0:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1275 device=0x1275 subvendor=0x0000 subdevice=0x0000
nvme0@pci0:0:4:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
nvme1@pci0:0:4:1: class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
nvme2@pci0:0:4:2: class=0x010802 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x0a0a subvendor=0x0000 subdevice=0x0000
virtio_pci0@pci0:0:5:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1000 subvendor=0x1af4 subdevice=0x0001
isab0@pci0:0:31:0: class=0x060100 rev=0x00 hdr=0x00 vendor=0x8086 device=0x7000 subvendor=0x0000 subdevice=0x0000
$ gpart show
=> 34 41942966 nda0 GPT (20G)
34 122 1 freebsd-boot (61K)
156 66584 2 efi (33M)
66740 2097152 3 freebsd-swap (1G)
2163892 39779108 4 freebsd-ufs (19G)
=> 40 440401840 nda1 GPT (210G)
40 440401840 1 freebsd-ufs (210G)
=> 40 314572720 nda2 GPT (150G)
40 314572720 1 freebsd-ufs (150G)