Solved can't restart VM in FreeBSD 13.1-RELEASE

_al

_al

Hi everyone,

This morning I upgraded FreeBSD 13.0-RELEASE to FreeBSD 13.1-RELEASE and now I can't restart VM.

What I doing:

I start VM with the command:
Code: 
sudo bhyve -c 8,sockets=2,cores=2,threads=2 -m 16G -H -w -S \
  -s 0,hostbridge \
  -s 3,ahci-cd,/neat/iso/Win10_21H1_Russian_x64.iso \
  -s 4,ahci-hd,/neat/bsd1/vms/w10.img \
  -s 5,virtio-net,tap0 \
  -s 7,passthru,216/0/0 \
  -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1200,wait \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  w10

Then, I connect to it:
Code: 
vncviewer freebsd:0

I do some work inside VM, include that which uses 'usb xhci pci' I passed thru to VM, there it is
Code: 
xhci3@pci0:216:0:0: class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014
subvendor=0x1912 subdevice=0x0014
vendor = 'Renesas Technology Corp.'
device = 'uPD720201 USB 3.0 Host Controller'
class = serial bus
subclass = USB

Then I quit VM and destroy it:
Code: 
sudo bhyvectl --destroy --vm=w10

After that I can't run it again:
Code: 
sudo bhyve -c 8,sockets=2,cores=2,threads=2 -m 16G -H -w -S \
  -s 0,hostbridge \
  -s 3,ahci-cd,/neat/iso/Win10_21H1_Russian_x64.iso \
  -s 4,ahci-hd,/neat/bsd1/vms/w10.img \
  -s 5,virtio-net,tap0 \
  -s 7,passthru,216/0/0 \
  -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1200,wait \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  w10

bhyve: Failed to map MSI-X table BAR on 216/0/0: Device busy
bhyve: failed to initialize BARs for PCI 216/0/0
device emulation initialization error: Device busy

I tried to kill VM again, but there is no more VM here (that's right):
Code: 
sudo bhyvectl --destroy --vm=w10

In FreeBSD 13.0-RELEASE I have no trouble with restart VM.

My rc.conf, loader.conf are the same as they were in FreeBSD 13.0-RELEASE:

Code: 
[lanin@freebsd /etc]$ cat rc.conf
...
kld_list="nvidia-modeset vmm nmdm"
cloned_interfaces="bridge0 tap0 tap1"
ifconfig_bridge0="addm igb0 addm tap0 addm tap1 up"

Code: 
[lanin@freebsd /boot]$ cat loader.conf
...
vmm_load="YES"
vmm_name="/boot/kernel/vmm.ko"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
pptdevs="134/0/0 216/0/0/0"
...

Can anyone help ?
Thanks
 
Last edited:
I can't directly help, but are you doing anything with audio inside the vm? If you search the forum there are some threads about being in a desktop environment, doing something with audio and then trying to log out. User can't log out because audio devices are being kept open.
In general, sounds like something is keeping a device open on that hub after you kill the vm.

Have you tried waiting a few minutes before restarting, just to see if it's something taking a while to clean up?
 
mer said:
are you doing anything with audio inside the vm?
Click to expand...
No, I have no audio inside VM, and do not try to use it.
The usb controller (I passed thru to VM) is used with the card readers to test smart cards.
mer said:
Have you tried waiting a few minutes before restarting, just to see if it's something taking a while to clean up?
Click to expand...
I tried to wait...and restart devfs service - no effect.

And , yes, I always disconnect all my readers from VM before I quit it.
 
  • Like
Reactions: mer
Thanks, I figured you were doing the right things.
As a data point, if you restart the vm but don't do the passthru, does it start? I know you can't test what you want, just verifying the passthru is the problem on the restart.

Is vncviewer still running when you shut down the vm? I know vncviewer should at least error when it loses the connection, but wanted to check that you are exiting vncviewer.
 
  • Thanks
Reactions: _al
And I got the same error even if I am not using this device in VM at all

mer said:
if you restart the vm but don't do the passthru, does it start?
Click to expand...
Yes! Now It works. But I need my device in VM
 
  • Like
Reactions: mer
Andrey Lanin said:
Yes, it is running when I quit VM. I'll try to quit it first.

Now, I'll reboot system and try all again.
Click to expand...
when I quit vncviewer while VM is up, VM's window disappeared, but VM still working...
And when I run vncviewer again, it connects to running VM. That's right.
When I quit VM while vncviewer is up, it quits automatically.

The problem is not gone - VM does not restart if it has device passed thru to it
I'll try to do some another experiments....
 
I tried to unload/load vmm.ko. It has been unloaded succsessfully (it means that bhyve's VM was destroyed before).
After vmm.ko has been loaded again, I tried to start VM, and got the same error...
 
  • Like
Reactions: mer
Thanks for the updates. I'm out of ideas on this, have you looked over at the FreeBSD bug database? Perhaps this is a bug somewhere especially since it worked correctly in 13.0 for you.
 
  • Thanks
Reactions: _al
So far I have only found this - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205549
I keep searching...
The thought is:
I have two passed thru devices, which differ only by PCI address:
Code: 
[lanin@freebsd ~]$ sudo pciconf -lc |grep -A3 -e "ppt0" -e "ppt1"
ppt0@pci0:134:0:0:    class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0014
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit 
    cap 11[90] = MSI-X supports 8 messages
--
ppt1@pci0:216:0:0:    class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0014
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit 
    cap 11[90] = MSI-X supports 8 messages

One of them I passed thru to VM-1, and other - to VM-2.
I start these VMs simultaneously.
My first VM has
Code: 
-s 7,passthru,216/0/0
and other -
Code: 
-s 7,passthru,134/0/0

Maybe the problem is somewhere here...
 
Found this - https://reviews.freebsd.org/D33739?id=100933

/usr/src/usr.sbin/bhyve/pci_passthru.c, L:451:
MSI-X table BAR.png
 
Not spam. Information. The best I can say is file a bug report, you have a repeatable configuration, you've done a lot of legwork investigating. All that makes it easier (not easy or trivial) for a developer to figure it out.
 
mer, thank you very much.
Honestly, I have never filed a bug report. I have to learn how it's done...
 
www.freebsd.org

FreeBSD Support

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

talks about filing a bug report and searching exisiting ones.
 
  • Thanks
Reactions: _al
I installed the bhyve+-0.1.0 and VMs began to restart well.
The error is gone.

But what is there in bhyve+ that eliminates this error ?
 
Thanks for the update. As to what's different? I have no idea.
If you go look at the git repo for it, this line sounds like it may apply:
Bjoern Zeeb's experimental patch for supporting non-standard mappings in the MSI-X table page is added.

Another thing is that it sounds like changes in bhyve+ get upstreamed to FreeBSD proper, so eventually the one in base should work.
It would be interesting to see if it just works in a 14-CURRENT system. But I'm not going to suggest that you update a running system.
 
  • Thanks
Reactions: _al
I have the same issue, the only way to restart my VM'S with passthru's is to reboot the server.
So If I understand correctly, the fix will make it to the portstree? Any Idea when?
 
You must log in or register to reply here.
Back
Top