Having an issue. I have 2 bhyve VM`s running. 1 VM - Linux, 2nd - FreeBSD 14.3-STABLE. and both have nVidia gpu passtrough but FreeBSD also have usb controller passed. Linux works fine, as its headless.
Im using uefi loader to get VNC for both VM`s.
My FreeBSD bhyve have this strange output from pciconf:
Which shows me its network card and also acts as display ?
My FreeBSD conf file:
My Host boot loader.conf
Host pciconf for USB on my main host.
Originally i wanted to pass 2 USB-A 2.0 ports and 2 USB-3 ports to FreeBSD VM but i just used whole xHCI controller.
dmesg of FreeBSD VM
When there was no nvidia kmod installed - i could use VNC but once i installed kmod, added nvidia-drm to rc.conf - cant do anything, Host and everything elses looses conection, tested with nvidia-modeset, same issue, changed to just nvidia - no changes.
I have 2 network nics. 1st nic is for Host, 2nd - for VM`s. I have not done nothing much for my nic appart adding up for my igb1.
My host /etc/rc.conf
Im using uefi loader to get VNC for both VM`s.
My FreeBSD bhyve have this strange output from pciconf:
Code:
pciconf -lv | grep -B3 display
vgapci0@pci0:0:2:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb2 subvendor=0x10de subdevice=0x11bd
vendor = 'NVIDIA Corporation'
device = 'GP107GL [Quadro P600]'
class = display
--
class = network
subclass = ethernet
vgapci1@pci0:0:7:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0xfb5d device=0x40fb subvendor=0x0000 subdevice=0x0000
class = displayMy FreeBSD conf file:
Code:
cat /vm/freebsd14/freebsd14.conf
loader="uefi"
# CPU
cpu=20
# MEMORY
memory=12G
# NETWORK
network0_type="virtio-net"
network0_switch="public"
# DISK
disk0_type="nvme"
disk0_name="disk0.img"
# GRAPHICS
#
passthru0="129/0/0=2:0"
passthru1="0/20/0=3:0"
graphics_res="1920x1200"
graphics="yes"
#graphics_res="1920x1080"
uuid="32b947dd-7126-11f0-a1ac-2cfda134338f"
network0_mac="58:9c:fc:03:0b:e7"My Host boot loader.conf
Code:
cat /boot/loader.conf
security.bsd.allow_destructive_dtrace=0
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
vmm_load="YES"
hw.vmm.enable_vtd="1"
pptdevs="130/0/0 129/0/0 0/20/0"Host pciconf for USB on my main host.
Code:
pciconf -lv | grep -B2 USB
ppt0@pci0:0:20:0: class=0x0c0330 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8d31 subvendor=0x1043 subdevice=0x85f6
vendor = 'Intel Corporation'
device = 'C610/X99 series chipset USB xHCI Host Controller'
class = serial bus
subclass = USB
--
ehci0@pci0:0:26:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8d2d subvendor=0x1043 subdevice=0x85f6
vendor = 'Intel Corporation'
device = 'C610/X99 series chipset USB Enhanced Host Controller'
class = serial bus
subclass = USB
--
ehci1@pci0:0:29:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8d26 subvendor=0x1043 subdevice=0x85f6
vendor = 'Intel Corporation'
device = 'C610/X99 series chipset USB Enhanced Host Controller'
class = serial bus
subclass = USB
[CODE]
As i see pci0:0:20:0 is my xHCI Host controller where my keyboard and hdd connected. I want to use my bhyve freebsd as main OS and i installed nvidia, xorg etc. but my monitor is in-active and with a minute or so im loosing internet connection to my host, rest of the vm`s and jails. have to do hard reboot.
Once i reboot - im fine with connections but have no access to VM`s graphical stuff nor VNS.
[CODE]
vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
archlinux default uefi 12 6144M 0.0.0.0:5900 Yes [1] Running (95268)
freebsd14 default uefi 20 12G 0.0.0.0:5901 Yes [2] Running (41075)dmesg of FreeBSD VM
Code:
dmesg
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.3-STABLE stable/14-n272106-bfbb93b6de6a GENERIC amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
VT(efifb): resolution 1920x1200
CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (2594.32-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
Features2=0xfede7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x140fb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP>
Structured Extended Features3=0x400<MD_CLEAR>
XSAVE Features=0x1<XSAVEOPT>
TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory = 12884901888 (12288 MB)
avail memory = 12446990336 (11870 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE BVAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 20 CPUs
FreeBSD/SMP: 20 package(s) x 1 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-31
Launching APs: 7 8 10 11 17 3 14 6 13 16 4 9 19 5 1 2 18 12 15
random: entropy device external interface
kbd1 at kbdmux0
Deprecated code (to be removed in FreeBSD 15): MIDI sequencer: no longer needed or used
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0xbfbd0000-0xbfbd001e
smbios0: Entry point: v2.1 (32-bit), Version: 2.8, BCD Revision: 2.8
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <BHYVE BVFACP >
acpi0: Power Button (fixed)
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 16777216 Hz quality 950
Event timer "HPET" frequency 16777216 Hz quality 350
Event timer "HPET1" frequency 16777216 Hz quality 350
Event timer "HPET2" frequency 16777216 Hz quality 350
Event timer "HPET3" frequency 16777216 Hz quality 350
Event timer "HPET4" frequency 16777216 Hz quality 350
Event timer "HPET5" frequency 16777216 Hz quality 350
Event timer "HPET6" frequency 16777216 Hz quality 350
Event timer "HPET7" frequency 16777216 Hz quality 350
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: no PRT entry for 0.2.INTA
pcib0: no PRT entry for 0.3.INTD
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0xc1000000-0xc1ffffff,0x800000000-0x80fffffff,0x810000000-0x811ffffff irq 10 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Wellsburg USB 3.0 controller> mem 0x812000000-0x81200ffff irq 10 at device 3.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
nvme0: <Generic NVMe Device> mem 0x812010000-0x812013fff,0xc2002000-0xc2003fff at device 4.0 on pci0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x2080-0x20bf mem 0xc2000000-0xc2001fff irq 18 at device 5.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 58:9c:fc:03:0b:e7
vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
000.000143 [ 452] vtnet_netmap_attach vtnet attached txq=1, txd=1024 rxq=1, rxd=512
vgapci1: <VGA-compatible display> mem 0xc2004000-0xc200407f,0xc0000000-0xc0ffffff at device 7.0 on pci0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
vmgenc0: <VM Generation Counter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 15.0.
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 4 on acpi0
uart3: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff pnpid PNP0900 on isa0
Timecounter "TSC-low" frequency 1297111786 Hz quality 1000
Timecounters tick every 10.000 msec
nvme0: temperature threshold not supported
nvme0: SET_FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000008 cdw11:00000000
nvme0: FEATURE NOT CHANGEABLE (01/0e) crd:0 m:0 dnr:0 p:1 sqid:0 cid:9 cdw0:0
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <bhyve-NVMe 1.0 NVME-4-0>
nda0: Serial Number NVME-4-0
nda0: nvme version 1.4
nda0: 102400MB (209715200 512 byte sectors)
Trying to mount root from ufs:/dev/nda0p2 [rw]...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Dual Console: Video Primary, Serial Secondary
uhub0: 21 ports with 21 removable, self powered
ugen0.2: <Corsair CORSAIR K70 RGB MK.2 Mechanical Gaming Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Corsair CORSAIR K70 RGB MK.2 Mechanical Gaming Keyboard, class 0/0, rev 2.00/3.24, addr 1> on usbus0
kbd2 at ukbd0
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device Western Digital My Passport 25E2 (0x1058:0x25e2)
ugen0.3: <Western Digital My Passport 25E2> at usbus0
umass0 on uhub0
umass0: <Western Digital My Passport 25E2, class 0/0, rev 3.10/40.05, addr 2> on usbus0
umass0: SCSI over Bulk-Only; quirks = 0x8000
umass0:1:0: Attached to scbus1
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <WD My Passport 25E2 4005> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 57584131453438324D594E31
da0: 400.000MB/s transfers
da0: 3815415MB (7813969920 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
ses0 at umass-sim0 bus 0 scbus1 target 0 lun 1
ses0: <WD SES Device 4005> Fixed Enclosure Services SPC-4 SCSI device
ses0: Serial Number 57584131453438324D594E31
ses0: 400.000MB/s transfers
ses0: SES Device
ugen0.4: <Samsung Flash Drive FIT> at usbus0
umass1 on uhub0
umass1: <Samsung Flash Drive FIT, class 0/0, rev 3.10/11.00, addr 3> on usbus0
umass1: SCSI over Bulk-Only; quirks = 0x4000
umass1:2:1: Attached to scbus2
da1 at umass-sim1 bus 1 scbus2 target 0 lun 0
da1: <Samsung Flash Drive FIT 1100> Removable Direct Access SPC-4 SCSI device
da1: Serial Number 0306919040007431
da1: 400.000MB/s transfers
da1: 61188MB (125313283 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
GEOM: da1: the secondary GPT header is not in the last LBA.
GEOM: iso9660/14_2_RELEASE_AMD64_DVD: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-0306919040007431: the secondary GPT header is not in the last LBA.
nvidia0: <Quadro P600> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
lo0: link state changed to UP
vtnet0: link state changed to UP
uhid0 on uhub0
uhid0: <Corsair CORSAIR K70 RGB MK.2 Mechanical Gaming Keyboard, class 0/0, rev 2.00/3.24, addr 1> on usbus0
Security policy loaded: MAC/ntpd (mac_ntpd)When there was no nvidia kmod installed - i could use VNC but once i installed kmod, added nvidia-drm to rc.conf - cant do anything, Host and everything elses looses conection, tested with nvidia-modeset, same issue, changed to just nvidia - no changes.
I have 2 network nics. 1st nic is for Host, 2nd - for VM`s. I have not done nothing much for my nic appart adding up for my igb1.
Code:
vm switch list
NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS
public standard vm-public 192.168.1.30/24 no - - igb1
Code:
cat /etc/rc.conf
clear_tmp_enable="YES"
hostname="kodasBSD"
ifconfig_igb0="inet 192.168.1.10 netmask 255.255.255.0 mtu 9000"
ifconfig_igb1="up mtu 9000"
defaultrouter="192.168.1.1"
ifconfig_igb0_ipv6="inet6 accept_rtadv"
gateway_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
# VM
vm_enable="YES"
vm_dir="zfs:zroot/vm"
vm_list="archlinux freebsd14"
vm_delay="3"
# JAILS
jail_enable="YES"
jail_parallel_start="YES"
jail_list="gitea postgres"