bhyve Can't get Win11 image to boot and have vnc port to connect

pathiaki

pathiaki

I need help debugging my file for creating a Win11 VM. It's refusing to start and connect to the VNC port of 5900.

vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
winguest default uefi 2 4G - No Stopped

My template file looks like this:

If you want to pull a graphical console, you'll need the UEFI loader,
# no matter what OS you're installing on the guest.

loader="uefi"
graphics="yes"
graphics_res="1920x1080"
graphics_port="5900"
xhci_mouse="yes"

# If not specified, cpu=n will give the guest n discrete CPU sockets.
# This is generally OK for Linux or BSD guests, but Windows throws a fit
# due to licensing issues, so we specify CPU topology manually here.

cpu=2
#cpu_sockets=1
#cpu_cores=2

# Remember, a guest doesn’t need extra RAM for filesystem caching--
# the host handles that for it. 4G is ludicrously low for Windows on hardware,
# but it’s generally more than sufficient for a guest.
memory=4G
#memory=16G

# put up to 8 disks on a single ahci controller. This avoids the creation of
# a new “controller” on a new “PCIe slot” for each drive added to the guest.

ahci_device_limit="8"

# e1000 works out-of-the-box, but virtio-net performs better. Virtio support
# is built in on FreeBSD and Linux guests, but Windows guests will need
# to have virtio drivers manually installed.

#network0_type="e1000"

network0_type="virtio-net"
network0_switch="public"

# bhyve/nvme storage is considerably faster than bhyve/virtio-blk
# storage in my testing, on Windows, Linux, and FreeBSD guests alike.

disk0_type="nvme"
disk0_name="disk0.img"
disk0_opts="maxq=16,qsz=8,ioslots=1,ser=ABCDEFGH"

# This gives the guest a virtual "optical" drive. Specifying disk1_dev=”custom”
# allows us to provide a full path to the ISO.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/zroot/bhyve/.iso/virtio-win-0.1.240.iso"

# windows expects the host to expose localtime by default, not UTC

utctime="no"

passthru0="4/0/0"

bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

uuid="bf99ed2f-51cb-11ef-ada9-00151739c0e4"
network0_mac="58:9c:fc:03:e8:f0"

The vm-bhyve.log:


Aug 11 10:37:21: initialising
Aug 11 10:37:21: [loader: uefi]
Aug 11 10:37:21: [cpu: 2]
Aug 11 10:37:21: [memory: 4G]
Aug 11 10:37:21: [hostbridge: standard]
Aug 11 10:37:21: [com ports: com1]
Aug 11 10:37:21: [uuid: 5ef4b0c4-5757-11ef-ada9-00151739c0e4]
Aug 11 10:37:21: [debug mode: no]
Aug 11 10:37:21: [primary disk: disk0.img]
Aug 11 10:37:21: [primary disk dev: file]
Aug 11 10:37:21: fatal; pci passthrough not supported on this system (no VT-d or amdvi)

The sysctl of amdvi:

sysctl hw.vmm.amdvi

hw.vmm.amdvi.domain_id: 0
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 0
hw.vmm.amdvi.count: 1

Does anyone see anything? Help is greatly appreciated.

Is there a debug mode that I can get more verbose (the log file seems to say there is) output? How do I turn it on?
 
The passthru is nvme (according to pciconf -lf)

nvme0@pci0:4:0:0: (I've now removed it)

The CPU is an AMD Ryzen 7700X

The latest attempt to boot from the log:

Aug 11 12:41:30: initialising
Aug 11 12:41:30: [loader: uefi]
Aug 11 12:41:30: [cpu: 2]
Aug 11 12:41:30: [memory: 4G]
Aug 11 12:41:30: [hostbridge: standard]
Aug 11 12:41:30: [com ports: com1]
Aug 11 12:41:30: [uuid: 5ef4b0c4-5757-11ef-ada9-00151739c0e4]
Aug 11 12:41:30: [debug mode: no]
Aug 11 12:41:30: [primary disk: disk0.img]
Aug 11 12:41:30: [primary disk dev: file]
Aug 11 12:41:30: initialising network device tap2
Aug 11 12:41:30: adding tap2 -> vm-public (public addm)
Aug 11 12:41:30: bring up tap2 -> vm-public (public addm)
Aug 11 12:41:30: booting
Aug 11 12:41:30: [bhyve options: -c 2 -m 4G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 8,hda,play=/dev/dsp,rec=/dev/dsp -U 5ef4b0c4-5757-
11ef-ada9-00151739c0e4]
Aug 11 12:41:30: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/zroot/bhyve/winguest/disk0.img,maxq=16,qsz=8,ioslots=1,ser=ABCDEFGH -s 5:0,ahci,cd:
/zroot/bhyve/.iso/virtio-win-0.1.240.iso -s 6:0,virtio-net,tap2,mac=58:9c:fc:08:17:e2 -s 7:0,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080 -s 8:0,xhci,tablet]
Aug 11 12:41:30: [bhyve console: -l com1,/dev/nmdm-winguest.1A]
Aug 11 12:41:30: starting bhyve (run 1)
Aug 11 12:41:30: bhyve exited with status 4
Aug 11 12:41:30: destroying network device tap2
Aug 11 12:41:31: stopped
 
That would remove the driver (1st line) for the disk repo of the VM (second line) and the disk io options for the repo. (Third line)

I beleive, from the log, it may be a problem with initiating the network.
 
pathiaki said:
That would remove the driver (1st line) for the disk repo of the VM (second line) and the disk io options for the repo. (Third line)

I beleive, from the log, it may be a problem with initiating the network.
Click to expand...
Nothing I see in this log points a network problem.
I saw that vm-bhyve set a COM console, which is absolutely useless with Windows.
 
I believe the console connection is to bhyve itself. (Don't quote me)

The present situation upon start up in the log is:

Aug 11 16:23:54: initialising
Aug 11 16:23:54: [loader: uefi]
Aug 11 16:23:54: [cpu: 2]
Aug 11 16:23:54: [memory: 4G]
Aug 11 16:23:54: [hostbridge: standard]
Aug 11 16:23:54: [com ports: com1]
Aug 11 16:23:54: [uuid: 24b7acb0-580e-11ef-ada9-00151739c0e4]
Aug 11 16:23:54: [debug mode: no]
Aug 11 16:23:54: [primary disk: ./disk0.img]
Aug 11 16:23:54: [primary disk dev: file]
Aug 11 16:23:54: initialising network device tap1
Aug 11 16:23:54: adding tap1 -> vm-public (public addm)
Aug 11 16:23:54: bring up tap1 -> vm-public (public addm)
Aug 11 16:23:54: booting
Aug 11 16:23:54: [bhyve options: -c 2 -m 4G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 8,hda,play=/dev/dsp,rec=/dev/dsp -U 24b7acb0-580e-
11ef-ada9-00151739c0e4]
Aug 11 16:23:54: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/zroot/bhyve/winguest/./disk0.img,maxq=16,qsz=8,ioslots=1,ser=ABCDEFGH -s 5:0,ahci,c
d:/zroot/bhyve/.iso/virtio-win-0.1.240.iso -s 6:0,virtio-net,tap1,mac=58:9c:fc:0c:86:4c -s 7:0,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080 -s 8:0,xhci,tablet]
Aug 11 16:23:54: [bhyve console: -l com1,/dev/nmdm-winguest.1A]
Aug 11 16:23:54: starting bhyve (run 1)
Aug 11 16:23:54: bhyve exited with status 4
Aug 11 16:23:54: destroying network device tap1
Aug 11 16:23:54: stopped

Everything that I'm finding on 'status 4' error points to a problem with the virtual switch. However, I've checked everything on that end. *shrug* I must not understand the issue.
 
See my response on another thread here.
I included a basic vm-bhyve config for a W11vm that is working on multiple bhyve systems.
Also, there is a link to apply the registry changes to bypass the TPM check at install time.


Yoru disk0_opts contains invalid options. According to vm-bhyve config, the following are valid options.
# Valid Options: direct,nocache,ro,sectorsize=logical[/physical]

To include those, you would probably need to use the bhyve_options:
Code: 
# bhyve_options
# any additional bhyve command line options
#
bhyve_options="-p 1:1"

Comment out the following lines in your config and see if it starts.
Code: 
disk0_opts="maxq=16,qsz=8,ioslots=1,ser=ABCDEFGH"
graphics_port="5900"
passthru0="4/0/0"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"
 
Well, it's not 100% but there's progress. CeXP: I'm getting the Status 4 error. Some Anon Guy: the VM has a status of running (but it's not) but not the VNC channel.

more *.log
Aug 21 12:37:54: generated static mac 58:9c:fc:08:dc:9a (based on 'winguest:0:1724258274:0')
Aug 21 12:38:00: initialising
Aug 21 12:38:00: [loader: uefi]
Aug 21 12:38:00: [cpu: 2]
Aug 21 12:38:00: [memory: 16G]
Aug 21 12:38:00: [hostbridge: standard]
Aug 21 12:38:00: [com ports: com1]
Aug 21 12:38:00: [uuid: b744978a-5fdb-11ef-877c-00151739c0e4]
Aug 21 12:38:00: [debug mode: no]
Aug 21 12:38:00: [primary disk: ./disk0.img]
Aug 21 12:38:00: [primary disk dev: file]
Aug 21 12:38:00: initialising network device tap1
Aug 21 12:38:00: adding tap1 -> vm-public (public addm)
Aug 21 12:38:00: bring up tap1 -> vm-public (public addm)
Aug 21 12:38:00: dynamically allocated port 5901 for vnc connections
Aug 21 12:38:00: booting
Aug 21 12:38:00: [bhyve options: -c 2 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U b744978a-5fdb-11ef-877c-00151739c0e4]
Aug 21 12:38:00: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/zroot/bhyve/winguest/./disk0.img -s 5:0,virtio-net,tap1,mac=58:9c:fc:08:dc:9a -s 6:
0,fbuf,tcp=0.0.0.0:5901,w=1920,h=1080,wait -s 7:0,xhci,tablet]
Aug 21 12:38:00: [bhyve console: -l com1,/dev/nmdm-winguest.1A]
Aug 21 12:38:00: [bhyve iso device: -s 3:0,ahci-cd,/zroot/bhyve/.iso/Win11_23H2_English_x64v2.iso,ro]
Aug 21 12:38:00: starting bhyve (run 1)
Aug 21 12:38:00: bhyve exited with status 4
Aug 21 12:38:00: destroying network device tap1
Aug 21 12:38:00: stopped

There's a few conflicting pages out there that number ahci as 0-7. Are you saying 8 is a viable address? Do I need to change the ahci setting or do I just add it to the bhyve_options?

Thank you both for the step up.
 
One thing I've run into, with WIndows, though it was Windows10, not 11, was that I needed in my vm-bhyve cofig

bhyve_options="-A"

See if that, while leaving those other options commented out, helps.
 
scottro said:
One thing I've run into, with WIndows, though it was Windows10, not 11, was that I needed in my vm-bhyve cofig

bhyve_options="-A"

See if that, while leaving those other options commented out, helps.
Click to expand...
It didn't. Thanks for the try. It seems that there's something amiss in that I'm getting a 'status 4' that there is no slot available. However, I don't know how to check available slots to remap.
 
So.... things are much better. However, I MAY have stumbled on a bug. I'm able to get things working with Some Anon's config file and I believed mine must have had a character in it as it wouldn't start my vm known as 'winguest'. I named the one I created with his template 'wintest' which worked perfectly. So, I removed winguest, verified there was nothing in /dev/vmm after the 'vm destroy winguest' command. I then tried to create winguest with the same templates. The same issue (didn't start and didn't have a port - also note that wintest started on port 5901 (not 5900)) Pretty confusing. So, now I create winguest2 and use the template and it WORKS. I then stop and remove it as well. I try to recreate and restart - NOPE. I create winguest3, it works fine.

I'm now at the stage of ignoring the bug for now. (I tried to find if port 5900 or others were open with netstat -a and got nothing.)

I got Windows to boot and went through the usual LabConfig to turn off everything it checks. My latest problem says that Windows requires 52GB and I'm only seeing 20GB despite having 300 GB configured so I have to figure out why it's not recognized and/or extend it (nothing seems obvious) at the install.

Thank you to the people who are assisting!
 
If you do not specify a disk size when creating the VM, the default will be 20G.
You can extend this using the truncate command.
Code: 
truncate -s +50G /path/to/diskX.img
would add 50G to the disk.
 
There were some really strange things happening including the aforementioned. I finally decided to clear everything with a reboot. (Not something I do with FreeBSD, but....) Things started working much better but still it seems to be tracking the names of previous machines.... I'm on winguest7 now. However, it came up with 300g as configured in the template file. :)

Also, the truncate command above will definitely be useful in the future. :D

Thank you all!
 
Latest state: I've got a VM up and ready to configure. However, it's not connecting to a network. I have all the windows configuration going up to the point where it is 'looking for a network'. Since there have been so many changes, time to post state and configs again:

vm-bhyve.log:

Aug 24 11:04:11: initialising
Aug 24 11:04:11: [loader: uefi]
Aug 24 11:04:11: [cpu: 2,sockets=1,cores=2,threads=1]
Aug 24 11:04:11: [memory: 16G]
Aug 24 11:04:11: [hostbridge: standard]
Aug 24 11:04:11: [com ports: com1]
Aug 24 11:04:11: [uuid: 9286e535-5ff9-11ef-877c-00151739c0e4]
Aug 24 11:04:11: [debug mode: no]
Aug 24 11:04:11: [primary disk: disk0.img]
Aug 24 11:04:11: [primary disk dev: file]
Aug 24 11:04:11: initialising network device tap0
Aug 24 11:04:11: adding tap0 -> vm-public (public addm)
Aug 24 11:04:11: bring up tap0 -> vm-public (public addm)
Aug 24 11:04:11: dynamically allocated port 5900 for vnc connections
Aug 24 11:04:11: booting
Aug 24 11:04:11: [bhyve options: -c 2,sockets=1,cores=2,threads=1 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 9286e535-5ff9-11ef-87
7c-00151739c0e4]
Aug 24 11:04:11: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/zroot/bhyve/winguest7/disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:02:41:4e -s 6:0
,fbuf,tcp=0.0.0.0:5900,w=1600,h=900 -s 7:0,xhci,tablet]
Aug 24 11:04:11: [bhyve console: -l com1,/dev/nmdm-winguest7.1A]
Aug 24 11:04:11: starting bhyve (run 1)

A look at IP Config: (DHCP knows about this subnet)

m-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 3e:96:d7:8e:01:5a
inet 192.168.0.224 netmask 0xffffffe0 broadcast 192.168.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 20000
groups: bridge vm-switch viid-4c918@
nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vmnet/winguest7/0/public
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:c7
inet6 fe80::5a9c:fcff:fe10:ffc7%tap0 prefixlen 64 scopeid 0x8
groups: tap vm-port
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 3213

winguest.conf:

loader="uefi"
graphics="yes"
xhci_mouse="yes"

cpu=2
cpu_sockets=1
cpu_cores=2
cpu_threads=1

memory=16G

# graphics_res
# This allows you to specify a resolution for the graphical console.
# Pleas note only the below options are supported
#
# Default: 800x600
# Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480
#
graphics_res="1600x900"

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
ahci_device_limit="8"

# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0.img"

# windows expects the host to expose localtime by default, not UTC
utctime="no"
uuid="9286e535-5ff9-11ef-877c-00151739c0e4"
network0_mac="58:9c:fc:02:41:4e"
 
Code: 
# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="virtio-net"
network0_switch="public"
to use network0_type="virtio-net" you will need to install the virtio drivers in windows. They can be found here.

To get it running without loading drivers, change that line to network0_type="e1000"
Windows already has drivers for the e1000 NIC built in.
After getting the system online using the e1000 setup, you can install the virt-io drivers. Then shutdown, change the config back to network0_type="virtio-net" and setup networking again.

The virt-io NIC should perform better than the e1000.
 
The e1000 drivers have been broken in windows 10 and "server" 2019 for ages now (and possibly in later versions too). you have to put the virtio-net drivers into the install image or attach an iso to the VM containing the virtio drivers to get that junk connected to a network.
This bullshit has also cost me several days a while ago, the e1000 driver *sometimes* actually gets a connection and even some traffic might be logged coming out of the VM, but after a few seconds the interface on the windows side goes completely dead. Of course no fix, not even a workaround despite *lots* of confirmed reports in those so-called microsoft "support forums"...
 
sko said:
The e1000 drivers have been broken in windows 10 and "server" 2019 for ages now (and possibly in later versions too). you have to put the virtio-net drivers into the install image or attach an iso to the VM containing the virtio drivers to get that junk connected to a network.
This bullshit has also cost me several days a while ago, the e1000 driver *sometimes* actually gets a connection and even some traffic might be logged coming out of the VM, but after a few seconds the interface on the windows side goes completely dead. Of course no fix, not even a workaround despite *lots* of confirmed reports in those so-called microsoft "support forums"...
Click to expand...

While there are a lot of posts / info on the buggy state of the e1000 driver, I have never encountered any of the issues.
That being said, I only ever use them for the install of Windows, and to download and install the virt-io drivers.
 
I had overlooked to test this e1000 virtual hardware. So, I tried it on a Windows 10 bhyve VM. It works at first glance.

For what it worths, I compared its speeds of download / upload on the internet with a virtio-net pseudo-hardware.

(e1000 compared to virtio-net) Download : +14 %, upload : -81 %

Well... :-/
 
Latest status: It's all working. Hooray!

The 'how did I get this working'

The e1000 switch worked. Thank you Emrion.

So, I switched to the network0_type="e1000".

I also added the following 3 lines so I could install the virtio-net once I booted off the e1000.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/zroot/bhyve/.iso/virtio-win-0.1.240.iso"

Those 3 lines will allow the user to install virtio-win by mounting it on the VMs CDROM device.

Once the machine is up and running, use the file manager to go to the VM's CDROM and run the <file>.exe to install it into the image of the VM.

After that's completed, you can switch over

network0_type="virtio-net"

I really want to write something up for start to finish that's more or less complete. It seems that bhyve changes frequently and I've looked up conflicting information leading to confusion. (It's actually a lot simpler than I thought but I had multiple conflicting documents stepping on my learning.)

Thank you all for the assistance.
 
You must log in or register to reply here.
Back
Top