Solved pcipassthru and tpm-Module

rfriemer

rfriemer

Hello community,

I have a problem, where I need our advice and help.

I have a FreeBSD 14.2 with bhyve and vm-bhyve running. On this machine I have a Windows11 VM with a tpm Module.

When I configure a pci passthrough of an USB-Controller. The VM doesn't boot. I had a Windows10 VM without the tpm-Module and there I could passthrough the USB-Controller. So I guess I have to reconfigure the VM, that I can use the tpm-Module and the USB-Controller.

Here are the configs.

/boot/loader.conf
Code: 
...
# TPM
tpm_load="yes"
...
# Bhyve PCI Passthrough
vmm_load="yes"
pptdevs="179/0/0"

The USB-Controller has the 179/0/0 ID.

Code: 
# vm passthru
...
ppt0       179/0/0      Yes          FL1100 USB 3.0 Host Controller

The Settings of the VM
Code: 
loader="uefi"

#debug="yes"

cpu=4
cpu_sockets=1
cpu_cores=2
cpu_threads=2

memory=16G

graphics="yes"
graphics_port="5911"
graphics_listen="127.0.0.1"
graphics_res="1920x1080"
graphics_wait="auto"

xhci_mouse="yes"

disk0_dev="file"
disk0_type="nvme"
disk0_name="windows11.img"

network0_type="virtio-net"
network0_switch="public"
network0_mac="58:9c:fc:0b:34:4d"

passthru0="179/0/0"

bhyve_options="-AHP -l tpm,passthru,/dev/tpm0 -l com1,/dev/nmdm0A"

utctime="no"

uuid="..."

In the info of the VM the passthrough-device isn't showing up. :(
Code: 
# vm info windows11
------------------------
Virtual Machine: windows11
------------------------
  state: running (34263)
  datastore: default
  loader: uefi
  uuid: ...
  cpu: 4
  cpu-topology: sockets=1, cores=2, threads=2
  memory: 16G
  memory-resident: 17195646976 (16.014G)

  console-ports
    com1: /dev/nmdm-windows11.1B
    vnc: 127.0.0.1:5911

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:0b:34:4d
    fixed-device: -
    active-device: tap0
    desc: vmnet/windows11/0/public
    mtu: 1500
    bridge: vm-public
    bytes-in: 204284733 (194.821M)
    bytes-out: 213326370 (203.443M)

  virtual-disk
    number: 0
    device-type: file
    emulation: nvme
    options: -
    system-path: /home/freebsd/bhyveVMs/windows11/windows11.img
    bytes-size: 94489280512 (88.000G)
    bytes-used: 32803415040 (30.550G)

  snapshots
    home/freebsd/bhyveVMs/windows11@2024-12-26-12:13:06 15.6G   Do. Dez. 26 12:13 2024

When the Line with the passthrough-Statement is active, the VM is stopping booting. When I comment this line out, then everything is fine.

See the bhyve-logs:
Code: 
Jan. 21 17:10:02: initialising
Jan. 21 17:10:02:  [loader: uefi]
Jan. 21 17:10:02:  [cpu: 4,sockets=1,cores=2,threads=2]
Jan. 21 17:10:02:  [memory: 16G]
Jan. 21 17:10:02:  [hostbridge: standard]
Jan. 21 17:10:02:  [com ports: com1]
Jan. 21 17:10:02:  [uuid: ...]
Jan. 21 17:10:02:  [debug mode: no]
Jan. 21 17:10:02:  [primary disk: windows11.img]
Jan. 21 17:10:02:  [primary disk dev: file]
Jan. 21 17:10:02: initialising network device tap0
Jan. 21 17:10:02: adding tap0 -> vm-public (public addm)
Jan. 21 17:10:02: bring up tap0 -> vm-public (public addm)
Jan. 21 17:10:02: booting
Jan. 21 17:10:02:  [bhyve options: -c 4,sockets=1,cores=2,threads=2 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -AHP -l tpm,passthru,/dev/tpm0 -l com1,/dev/nmdm0A -U c430fa0f-3ad1-11e9-a196-00241dcd6d30 -S]
Jan. 21 17:10:02:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/home/freebsd/bhyveVMs/windows11/windows11.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:34:4d -s 6:0,passthru,179/0/0 -s 7:0,fbuf,tcp=127.0.0.1:5911,w=1920,h=1080 -s 8:0,xhci,tablet]
Jan. 21 17:10:02:  [bhyve console: -l com1,/dev/nmdm-windows11.1A]
Jan. 21 17:10:02: starting bhyve (run 1)
Jan. 21 18:22:45: initialising
Jan. 21 18:22:45:  [loader: uefi]
Jan. 21 18:22:45:  [cpu: 4,sockets=1,cores=2,threads=2]
Jan. 21 18:22:45:  [memory: 16G]
Jan. 21 18:22:45:  [hostbridge: standard]
Jan. 21 18:22:45:  [com ports: com1]
Jan. 21 18:22:45:  [uuid:...]
Jan. 21 18:22:45:  [debug mode: no]
Jan. 21 18:22:45:  [primary disk: windows11.img]
Jan. 21 18:22:45:  [primary disk dev: file]
Jan. 21 18:22:45: initialising network device tap0
Jan. 21 18:22:45: adding tap0 -> vm-public (public addm)
Jan. 21 18:22:45: bring up tap0 -> vm-public (public addm)
Jan. 21 18:22:45: booting
Jan. 21 18:22:45:  [bhyve options: -c 4,sockets=1,cores=2,threads=2 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -AHP -l com1,/dev/nmdm0A -U c430fa0f-3ad1-11e9-a196-00241dcd6d30 -S]
Jan. 21 18:22:45:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/home/freebsd/bhyveVMs/windows11/windows11.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:34:4d -s 6:0,passthru,179/0/0 -s 7:0,fbuf,tcp=127.0.0.1:5911,w=1920,h=1080 -s 8:0,xhci,tablet]
Jan. 21 18:22:45:  [bhyve console: -l com1,/dev/nmdm-windows11.1A]
Jan. 21 18:22:45: starting bhyve (run 1)
Jan. 21 18:59:49: initialising
Jan. 21 18:59:49:  [loader: uefi]
Jan. 21 18:59:49:  [cpu: 4,sockets=1,cores=2,threads=2]
Jan. 21 18:59:49:  [memory: 16G]
Jan. 21 18:59:49:  [hostbridge: standard]
Jan. 21 18:59:49:  [com ports: com1]
Jan. 21 18:59:49:  [uuid: ...]
Jan. 21 18:59:49:  [debug mode: no]
Jan. 21 18:59:49:  [primary disk: windows11.img]
Jan. 21 18:59:49:  [primary disk dev: file]
Jan. 21 18:59:49: initialising network device tap0
Jan. 21 18:59:49: adding tap0 -> vm-public (public addm)
Jan. 21 18:59:49: bring up tap0 -> vm-public (public addm)
Jan. 21 18:59:49: booting
Jan. 21 18:59:49:  [bhyve options: -c 4,sockets=1,cores=2,threads=2 -m 16G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -AHP -l tpm,passthru,/dev/tpm0 -l com1,/dev/nmdm0A -U c430fa0f-3ad1-11e9-a196-00241dcd6d30]
Jan. 21 18:59:49:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/home/freebsd/bhyveVMs/windows11/windows11.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:34:4d -s 6:0,fbuf,tcp=127.0.0.1:5911,w=1920,h=1080 -s 7:0,xhci,tablet]
Jan. 21 18:59:49:  [bhyve console: -l com1,/dev/nmdm-windows11.1A]
Jan. 21 18:59:49: starting bhyve (run 1)

So, my question is, how can I fix this?
 
why not use the -l option for bhyve(8) which supports (software/emulated) tpm devices? There's also sysutils/swtpm to emulate TPM for VMs.
Or just don't use TMP at all, it's mostly broken by design and snake-oil anyways...
 
Hello sko,

as I know, you have to use a tpm-Module for Windows11 you. So, this is the reason why I'm implemented it.

And in my vm config I use the -l option.

An other point is, that I also tried to include the passthru in the bhyve_option line, but this also didn't worked. :(
 
rfriemer said:
Code: 
passthru0="179/0/0"
Click to expand...
It's missing the assignment for the VM's slot. I use these to passthrough a SAS/SATA and USB3 controller to a VM, they are on 2/0/0 and 7/0/0 on the host:
Code: 
passthru0="2/0/0=2:0"
passthru1="7/0/0=7:0"
On the host:
Code: 
# pciconf -l | grep ppt
ppt0@pci0:7:0:0:        class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x2142 subvendor=0x3142 subdevice=0x1b21
ppt1@pci0:2:0:0:        class=0x010700 rev=0x02 hdr=0x00 vendor=0x1000 device=0x0072 subvendor=0x15d9 subdevice=0x0400
On the guest VM:
Code: 
# pciconf -l
hostb0@pci0:0:0:0:      class=0x060000 rev=0x00 hdr=0x00 vendor=0x1275 device=0x1275 subvendor=0x0000 subdevice=0x0000
mps0@pci0:0:2:0:        class=0x010700 rev=0x02 hdr=0x00 vendor=0x1000 device=0x0072 subvendor=0x15d9 subdevice=0x0400
virtio_pci0@pci0:0:4:0: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1001 subvendor=0x1af4 subdevice=0x0002
virtio_pci1@pci0:0:5:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1000 subvendor=0x1af4 subdevice=0x0001
xhci0@pci0:0:7:0:       class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x2142 subvendor=0x3142 subdevice=0x1b21
isab0@pci0:0:31:0:      class=0x060100 rev=0x00 hdr=0x00 vendor=0x8086 device=0x7000 subvendor=0x0000 subdevice=0x0000
Notice the mps(4) and xhci(4) devices on 0:2:0 and 0:7:0.
 
rfriemer said:
And in my vm config I use the -l option.

An other point is, that I also tried to include the passthru in the bhyve_option line, but this also didn't worked. :(
Click to expand...

Sorry, I completely misread your question - I thought you want to pass through an USB-controller to which you attached a USB-TPM-module.
So the problem is using passthrough AND the '-l' option combined?


Regarding Windows 11 and TPM: you can circumvent that arbitrary requirement (as well as others) with some simple registry changes. Not using Win11 anywhere though, so no idea if that still applies... (but a quick search gives plenty of results)
 
hellp sko ,

sorry for my delayed answers.

no, no, no... The TPM-Module is attached directly to the Mainboard to it's Port. The USB-Controller is a separate PCI Card.

I want to have a simply installed Windows11 I know it since Windows for Workgroups. So quite a little while.

fTPM is also possible, but I have a TPM-Module, and I want to use this.

Robert
 
SirDice said:
It's missing the assignment for the VM's slot. I use these to passthrough a SAS/SATA and USB3 controller to a VM, they are on 2/0/0 and 7/0/0 on the host:
Code: 
passthru0="2/0/0=2:0"
passthru1="7/0/0=7:0"
On the host:
Code: 
# pciconf -l | grep ppt
ppt0@pci0:7:0:0:        class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x2142 subvendor=0x3142 subdevice=0x1b21
ppt1@pci0:2:0:0:        class=0x010700 rev=0x02 hdr=0x00 vendor=0x1000 device=0x0072 subvendor=0x15d9 subdevice=0x0400
On the guest VM:
Code: 
# pciconf -l
hostb0@pci0:0:0:0:      class=0x060000 rev=0x00 hdr=0x00 vendor=0x1275 device=0x1275 subvendor=0x0000 subdevice=0x0000
mps0@pci0:0:2:0:        class=0x010700 rev=0x02 hdr=0x00 vendor=0x1000 device=0x0072 subvendor=0x15d9 subdevice=0x0400
virtio_pci0@pci0:0:4:0: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1001 subvendor=0x1af4 subdevice=0x0002
virtio_pci1@pci0:0:5:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1000 subvendor=0x1af4 subdevice=0x0001
xhci0@pci0:0:7:0:       class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1b21 device=0x2142 subvendor=0x3142 subdevice=0x1b21
isab0@pci0:0:31:0:      class=0x060100 rev=0x00 hdr=0x00 vendor=0x8086 device=0x7000 subvendor=0x0000 subdevice=0x0000
Notice the mps(4) and xhci(4) devices on 0:2:0 and 0:7:0.
Click to expand...
Hello SirDice ,

I will try your suggestions. Perhaps this will solve the problem.

I will give you a feedback.

Robert
 
That's Funny!!!!

SirDice : I have tested yopur advice

Code: 
passthru0="179/0/0=179:0"

But the VM didn't booted at all.

I've checked the pci.

Code: 
# pciconf -l | grep ppt
ppt0@pci0:179:0:0:      class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x1b73 device=0x1100 subvend
or=0x1b73 subdevice=0x1100

But it didn't booted. So I have looked at the configuration and suddenly I had an idea!!!! Rember my configuration?

Code: 
vm configure windows11
...
passthru0="179/0/0"

bhyve_options="-AHP -l tpm,passthru,/dev/tpm0 -l com1,/dev/nmdm0A"
...

Why do I need a serial Port on a WINDOWS-VM?????????

So I have removed the serial Port and activated the PCI USB-Controller and voilà the VM is booting AND I have a USB-Controller!!!!

So the coonfiguration for my Windows11 VM is as follows.

Code: 
loader="uefi"

#debug="yes"

cpu=4
cpu_sockets=1
cpu_cores=2
cpu_threads=2

memory=16G

graphics="yes"                  
graphics_port="5911"            
graphics_listen="127.0.0.1"      
graphics_res="1920x1080"        
graphics_wait="auto"            
                                 
xhci_mouse="yes"                
                                 
disk0_dev="file"                
disk0_type="nvme"                
disk0_name="windows11.img"      
                                 
network0_type="virtio-net"      
network0_switch="public"
network0_mac="58:9c:fc:0b:34:4d"

passthru0="179/0/0"

bhyve_options="-AHP -l tpm,passthru,/dev/tpm0"

utctime="no"

uuid="..."

Thank you, SirDice and sko
 
So, after a while I'm back with this issue.

Unfortunately the Windows VM is not booting when the PCI-USB-Controller is passthru in the config.

So, I have to look. :(

Oh, I have forgotten to mention, that the PCI-Passthru is working on Linux, so it must be a Windows Issue. :(
 
Ok, just to finish this Thread.

At the end, I got the PCI-Passthru running with Windows11, but with two changes:
  1. The TPM-Module is not in the VM. Windows11 is still running, but I can't say from how long.
  2. I have configured only 2 Processors with no threads.
The Setup is very fragile. When you try to change something then the VM could hang at the next reboot.

Thanks for all the help.

Robert
 
You must log in or register to reply here.
Back
Top