bhyve Trying to apply the patches to enable the passthru of my Intel GPU on a bhyve VM

[ziomario]

Hello to everyone.

I'm trying to apply the corvin's patches related to the passthrough of a gvt-d intel GPU. Corvin some sime ago gave me some scripts that I have always used to apply the patches needed to passthru of my nvidia GPU to a linux vm. But today I've realized that inside his repository there are the patches necessary to passthru also my Intel GPU. So,I tried to apply those patches,that you can find here :

https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/gvt-d

and this is what I did :

# ./build_branch.sh origin/phab/corvink/gvt-d --without-bhf --verbose

everything went good until this point :

# ./build_branch.sh origin/phab/corvink/gvt-d --without-bhf --verbose

cc -O2 -pipe -fno-common -I/usr/src/usr.sbin/bhyve/../../contrib/lib9p -I/usr/src/sys -DINET -DINET6
-DNETGRAPH -I/usr/src/sys/dev/e1000 -I/usr/src/sys/dev/mii -I/usr/src/sys/dev/usb/controller -fPIE
-g -gz=zlib -std=gnu99 -Wno-format-zero-length -fstack-protector-strong -Wsystem-headers -Werror -Wa
ll -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-un
used-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -
Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-ad
dress-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Qunused-arguments  
-pie   -o bhyve.full acpi_device.o atkbdc.o acpi.o audio.o bhyvegc.o bhyverun.o block_if.o bootrom.o
 config.o console.o ctl_util.o ctl_scsi_all.o e820.o fwctl.o gdb.o hda_codec.o inout.o ioapic.o kern
emu_dev.o mem.o mevent.o mptbl.o net_backends.o net_utils.o pci_ahci.o pci_e82545.o pci_emul.o pci_h
da.o pci_fbuf.o pci_gvt-d.o pci_hostbridge.o pci_irq.o pci_lpc.o pci_nvme.o pci_passthru.o pci_virti
o_9p.o pci_virtio_block.o pci_virtio_console.o pci_virtio_input.o pci_virtio_net.o pci_virtio_rnd.o
pci_virtio_scsi.o pci_uart.o pci_xhci.o pctestdev.o pm.o post.o ps2kbd.o ps2mouse.o qemu_fwcfg.o rfb
.o rtc.o smbiostbl.o sockstream.o task_switch.o uart_emul.o usb_emul.o usb_mouse.o virtio.o vga.o vm
genc.o xmsr.o spinup_ap.o iov.o vmm_instruction_emul.o -L/usr/obj/usr/src/amd64.amd64/lib/libvmmapi
-lvmmapi -L/usr/obj/usr/src/amd64.amd64/lib/libmd -lmd -L/usr/obj/usr/src/amd64.amd64/lib/libnv -ln
v -L/usr/obj/usr/src/amd64.amd64/lib/libthr -L/usr/obj/usr/src/amd64.amd64/lib/libz -lz -L/usr/obj/u
sr/src/amd64.amd64/lib/libutil -lutil -L/usr/obj/usr/src/amd64.amd64/lib/libsbuf -lsbuf -L/usr/obj/u
sr/src/amd64.amd64/lib/libcam -lcam -L/usr/obj/usr/src/amd64.amd64/lib/lib9p -l9p -L/usr/obj/usr/src
/amd64.amd64/lib/libcasper/libcasper -lcasper -L/usr/obj/usr/src/amd64.amd64/lib/libcasper/services/
cap_pwd -lcap_pwd -L/usr/obj/usr/src/amd64.amd64/lib/libcasper/services/cap_grp -lcap_grp -L/usr/obj
/usr/src/amd64.amd64/lib/libnetgraph -lnetgraph -L/usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto
-lcrypto -lpthread  
ld: error: undefined symbol: fspacectl
>>> referenced by block_if.c:345 (/usr/src/usr.sbin/bhyve/block_if.c:345)
>>> block_if.o:(blockif_thr)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

I don't know the reason of this error. I don't know if I've used a wrong procedure to apply those patches. I don't know if there is some error inside the code. Someone more experienced than me,can help me to understand what to do ? Below I have added all the scripts that Corvin gave to me and that I have always used to patch bhyve to allow the passthru of the NVidia GPU,not of the intel GPU. If you want to give a look at the code,you could understand where is the error. Very thanks.

build.sh : https://pastebin.ubuntu.com/p/H4fCTQk5Zd/
build_branch.sh : https://pastebin.ubuntu.com/p/zMNSdqDBbQ/
setup_git.sh : https://pastebin.ubuntu.com/p/gNtbK7Qfzz/

 

[yuABXBM]

Hi! Did you get further with this ? Looking to do the same.

 

[monwarez]

What version of CURRENT are you using in the source tree ? fspacectl syscall seems to appears in 1400032

 

[ziomario]

Did you get this error ?

bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory.
GPU passthrough might not work properly.
bhyve: gvt_d_setup_opregion: Unable to get OpRegion base and length
bhyve: gvt_d_init: Unable to setup OpRegion
device emulation initialization error: Operation not supported by device

I've found that it can be fixed with a fresh reinstallation of FreeBSD 13.1 + re applying every patch from the corvin's repo,using this command :

./build_branch.sh origin/phab/corvink/releng/13.1 --without-bhf --verbose

Tell me if you will encounter the same problem. Corvin said that his gvt-d patch has been written for FreeBSD 14 and it may or not may work on 13.1. For me it works only if I reinstall FreeBSD 13.1 from scratch and later I apply every patch of Corvin with the command above. I mean,if you apply the corvins patches on the installation that you are currently running you will get that error. Anyway,I think that copying some specific bhyve file from a fresh and patched 13.1 to the 13.1 that you are currently running could help you to passthru the gvt-d and the nvidia gpus at the same time. I'm still working on this.

 

[astyle]

bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory.

It may help to know what Graphics Stolen Memory even is... I'd start there for debugging THIS stuff.

 

[ziomario]

This kind of information for a newbie like me is a little unuseful and a wasting of time 'cause the complexity of the concepts behind what I experience directly,since I haven't any solid background in computer science. Anyway,if someone explains this to me,I will try to understand. But you know,on internet rarely the experienced users explains the theory behind the errors. Usually they try to solve the problem with short,fast and pratical examples. And this is what I'm looking for. Without a solid background,I try to learn by doing.

 

[zirias@]

It may help to know what Graphics Stolen Memory even is... I'd start there for debugging THIS stuff.

Some (typically integrated) GPUs use a part of the main RAM as graphics RAM, you can sometimes configure how much of it in BIOS settings. So I'd assume it's about that.

 

[ziomario]

Some (typically integrated) GPUs use a part of the main RAM as graphics RAM, you can sometimes configure how much of it in BIOS settings. So I'd assume it's about that.

so ok. but is this info useful for us ? I don't think so much. I've found the workaround with my scientific approach of finding the wrong variables isolating the variables one by one,doing a lot of tests...basing the choice on the empirical tests that I did and getting the informations from my experience and..asking and comparing again and again...you don't have idea...mine is a tedious approach. Everytime I should almost reinvent the wheel. All of this because I need to compensate the lack of knowledge I have. I would deserve a medal for all the time I dedicate to FreeBSD and the computer science....So,the practical approach won over the theoretical one.

 

[astyle]

Most of the time, solid knowledge of theory really helps in practice.

As an unrelated example, a solid knowledge of Directed Acyclic Graphs is what helps me avoid cyclic dependencies when compiling ports with EVERYTHING enabled. And if I run into issues, I know to backtrack, along which dependency lines, and why I pick these (and not others). The same theoretical foundation is also behind the design of firewall rules... and git branching.

 

[yuABXBM]

Mega disclaimer, I'm *planning* to jump ship and install FreeBSD on my main laptop. -as such: quite the noob in all this.

But, I had to change the defaults for the hypervisor with kvm so it allowed higher memlock. Needed to allocate memory for vga-passthrough (bad memory error on the default limit).

Are there similar limits with bhyve ?

 

[astyle]

Mega disclaimer, I'm *planning* to jump ship and install FreeBSD on my main laptop. -as such: quite the noob in all this.

But, I had to change the defaults for the hypervisor with kvm so it allowed higher memlock. Needed to allocate memory for vga-passthrough (bad memory error on the default limit).

Are there similar limits with bhyve ?

Take a look at FreeBSD's laptop wiki... the big hang-up seems to be that you gotta have Intel-based wi-fi as an m.2 card. However, if you build a box with aftermarket parts, there shouldn't be much of an issue running FreeBSD. =_=

 

[Phishfry]

Sorry to threadjack but I am an ardent Atheros supporter.

the big hang-up seems to be that you gotta have Intel-based wi-fi as an m.2 card

There is an Atheros card for that. We have a single M.2 card that works.

Qualcomm Atheros QCNFA222 - TechInfoDepot

"NFA222", "250-02471-020", "E85792" and "1432-01" (08/2014)

en.techinfodepot.shoutwiki.com

Atheros QCNFA222 AR9462 M.2 NGFF 802.11N 300Mbps Bluetooth WIFI 2.4Ghz 5Ghz | eBay

Find many great new & used options and get the best deals for Atheros QCNFA222 AR9462 M.2 NGFF 802.11N 300Mbps Bluetooth WIFI 2.4Ghz 5Ghz at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[yuABXBM]

Appreciate the suggestions, but I was mainly addressing this error:

bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory.

Intel gvt-d produce the same error on hypervisors such as kvm/qemu with the default memlock limits. Does bhyve have some similar limitation(s) on what a VM can request from the hypervisor ? To mitigate "out of bounds" type vulnerabilities? mmap? Full passthrough of vga requires -a lot- more space to be allocated

But as I said, Im quite the noob and all this might be utterly and embarrassingly off

 

[ziomario]

Appreciate the suggestions, but I was mainly addressing this error:



Intel gvt-d produce the same error on hypervisors such as kvm/qemu with the default memlock limits. Does bhyve have some similar limitation(s) on what a VM can request from the hypervisor ? To mitigate "out of bounds" type vulnerabilities? mmap? Full passthrough of vga requires -a lot- more space to be allocated

But as I said, Im quite the noob and all this might be utterly and embarrassingly off

Do you want to fix it or not ? I've explained how.

 

[yuABXBM]

Do you want to fix it or not ? I've explained how.

I wouldnt mind being allowed to ask 2 questions to understand whats causing the problem in the first place before I get ultimatums. As I mentioned, I'm a new user here and have no idea what I did to offend you so much that it warrants this kind of behavior. What am I missing?

 

[ziomario]

Your approach is good. I mean,asking why,the reasons behind the errors. I'm in contact with a bhyve developer and I tried to ask the same to him and he tried to explain.

origin/phab/corvink/gvt-d is very old. It’s likely that it’s outdated.
On which FreeBSD version would you like to run bhyve? 13.1 or 13.0?
I know upgrading a system has some risk.
Nevertheless, I’m alone with my GVT-d patches.
So, maintaining the GVT-d patches is a lot of work.
I’m not willing to support old FreeBSD versions. It’s just not doable.
If you like to use 13.0 the only branch you could use is:

origin/phab/corvink/13.0/nvidia-wip

I’m not sure if this branch is up to date. Please switch to 13.1 and use:

origin/phab/corvink/releng/13.1

Btw: I’m very slowly making progress. So far, AMD and Intel GPU passthrough is partly upstream.

While AMD GPU passthrough is fully supported by bhyve, it’s not supported by OVMF yet. (Windows works, Linux/BSD not)

Intel GPU passthrough only works for Linux yet. (Windows and pre OS graphics aren’t supported yet)

The issue is unrelated to my patches.

They are related to your include files. They don’t match the bhyve version.

If you compile a 13.1 bhyve you should use the include files of 13.1.

The easiest way to make sure you have the correct include files is to use the same FreeBSD version.

My gvt-d patches are 14.0-CURRENT. So, you should use 14.0-CURRENT. Nevertheless, this might not work.

14.0-CURRENT changes a lot, so maybe it’s too new.

I’m going to rebase my gvt-d branch. It’ll be updated this week.

Nevertheless, if you use 13.1 you should use:

origin/phab/corvink/releng/13.1

It includes all of my gvt-d patches.


First of all “bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory” is what it says.

It’s just a warning. Normally, it should hurt. It might get important on future GPU generations.

The other messages are real errors. They say that bhyve is unable to detect the OpRegion and GSM address.

This could have two reasons:

1. Your GPU has not OpRegion/GSM (that’s the case when you boot your GPU in CSM mode)
2. Your using an unsupported GPU

You told me that you don’t use CSM. So, please send me the device id of you GPU. I’ll check if it’s supported or not.

The device id can be read by:

pciconf -r -h pci0:0:2:0 0x02

 

[yuABXBM]

Thanks for clearing it up and elaborating on the issue. Appreciated!

I will attempt what you suggested this weekend.

 

[yuABXBM]

I'm in contact with a bhyve developer and I tried to ask the same to him and he tried to explain.

Thanks for quoting that explanation / the hint. That led me to this reference manual of the intel open source reference manual (although a bit old), which I'm chewing on right now. A bit blank on how bhyve, video oprom and the actual opregion is filled, but I think it was premature to dedicate my weekend. I need to wrap my head around this one way or another first (and I'll stop moving this thread more and more off topic until that. )

 

[ziomario]

if you have the skill to understand that,ok. I don't have such kind of skills. Usually I fix the thecnical problems in a pratical way,reading how to do from short and concise tutorials that go straight to the point.

 

[yuABXBM]

if you have the skill to understand that,ok. I don't have such kind of skills. Usually I fix the thecnical problems in a pratical way,reading how to do from short and concise tutorials that go straight to the point.

I definitely wont understand, but I might grasp something.

A short recipe running on my system without understanding why, might as well run some proprietary system with some sort of accountability for that problem.

But in reality, might use the tutorial as a starting base for trying to untangling something at least

 

[ziomario]

I definitely wont understand, but I might grasp something.

A short recipe running on my system without understanding why, might as well run some proprietary system with some sort of accountability for that problem.

But in reality, might use the tutorial as a starting base for trying to untangling something at least

It's not a better idea to find an easier and less technical tutorial ? and maybe to compare the differences that there are between different FreeBSD versions / installations with and without the bhyve patches ? if you follow a theoretical approach without having strong skills you will end making nothing.

 

[yuABXBM]

It's not a better idea to find an easier and less technical tutorial ?

Tutorial for health: "eat fruit daily".

Some find that easier with why/how etc that tutorial should be followed at all etc.

and maybe to compare the differences that there are between different FreeBSD versions / installations with and without the bhyve patches ?

tutorial v1.0: "eat 1 fruit daily"
tutorial v1.1: "eat more than 1 fruit daily".

The difference is easy to spot. But I'll give you an practical example:

if you follow a theoretical approach without having strong skills you will end making nothing.

See, some find it sane and easier to questioning claims instead of blindly eating 2 bananas without any thought: do you or dont you want to be healthy might your way. And thats ok

 

[ziomario]

did you have a degree in computer science ?

 

[astyle]

if you follow a theoretical approach without having strong skills you will end making nothing.

Now that I disagree with - and I do have a Computer Science degree. Theoretical approach allows you to understand what a tool is good for. For example, you can't use ping to figure out the pathway to the host - gotta use traceroute for that.

'Strong skills' are meaningless without sound theory backing them.

 

[ziomario]

I agree. But what to do when you have chosen to do something else in your life but you like to "play" with computers ? What to do when you haven't a solid background ? What to do when you don't understand well much of the theory behind the tasks that you accomplish ?