Solved Trying to understand why only some kind of vmm.ko module allows to passthru my nvidia GPU to a WIndows 11 vm

Topic here is solved, OP got explanation and solution how to do this. That ought to be enough.

I do believe bileslav didn't provide the module. There would be really no point to hide this, it's a clever, dirty hack to fool the pass-through. And ziomario you did admit yourself you are not 100% sure you have it from him:

ziomario said:
[FONT=monospace]bileslav[/FONT] : there still could be a small chance that I'm not using the vmm.ko that you have given to me.
Click to expand...

To me personally it matters because I've seen somebody being bashed here on forums for something he didn't do. Frankly past few years community here around this forum is not to my liking and I do avoid it more nowadays.

Now if the official module was binary patched and shared build id of the module would be the same as one of the id from the official modules. If the source code was customized and module compiled then id would be different. Not that it matters as one can do nothing with either information.

Hence again, kudos to anonymous person who shared the hack.
 
ziomario said:
Ok. But what's the earning that nVidia has by releasing the drivers for FreeBSD ?
Click to expand...
My understanding is "one of the contributions to opensource world".
Or a kind of "noblesse oblige. On other (evil) word, not to be beaten.
And shrink of BSD 2 clause (including compatible) licensed project is a disaster for companies.
GPL forces delivertive and shipped works to be open source. This is most handy fot corporates to disclose their softwares, as if their competitors use the codes, the competitors must disclose what they modified as of GPL.
But BSD 2 clause license does not force disclosure, even for "we used the code". So codes of FreeBSD is very handy to reuse for corporates.
 
_martin said:
you did admit yourself you are not 100% sure you have it from him
Click to expand...

100% I've got the module from him ; 2% even if I've got the module from him,I already had a module like his WITHOUT knowing it,because before using his module,I wasn't able to passthru the GPU... As you can see,this event is very unluckily.
 
This thread is too long, good information is being lost due to "side quests". Rereading this is a bit of pain.

What bileslav mentioned here is important. He mentioned the github repositories. I googled around and found this article: GPU passthrough for bhyve on FreeBSD 14.0-RELEASE
That circles back to bileslav github reference and points exactly here: x86.c#L54

So that is the origin of this idea of modifying the string.
 
Nice,it seems that you found the origin. What I still don't understand is why the passthru of my GPU under Windows never worked for me. I mean,I use the Corvin's code and patches from years,I have even helped him a lot to test the patches he created to allow the passthru of my GPU under Linux,I'm still using his code now,but the passthru for Windows never worked for me. I want to also add that I have helped Bileslav to pass his GPU under Linux,because it didn't work for him and thanks to me,it started working. And he gave me all the bhyve files he had to help me to pass my GPU under Windows. So,it seems that inside the Corvin's code there is something that does not work. It seems that if you get the code from him two separate phenomenons could happens :

1) you see your nvidia GPU passed under Linux

OR

2) you see your nvidia GPU passed under Windows

Bileslav told me that he was able to use the same bhyve executable to pass his GPU under Linux and Windows. His file never worked for me. It seems that the patches implemented for Linux interferes with the passthru of the GPU for Windows (at least for me). Infact I'm actually using two different bhyve executables : bhyve-win and bhyve-lin.
 
ziomario said:
Bileslav told me that he was able to use the same bhyve executable to pass his GPU under Linux and Windows. His file never worked for me. It seems that the patches implemented for Linux interferes with the passthru of the GPU for WIndows (at least for me). Infact I'm actually using two different bhyve executables : bhyve-win and bhyve-lin.
Click to expand...

If you look into that Beckhoff repository, there is couple of commits where they apparently tried different things. Consequentially there may be various versions with different behaviour...
 
Tried to boot the Windows vm using bhyve-lin,that's,instead,able to pass the nvidia GPU to a Linux vm :

Code: 
/usr/sbin/./bhyve-lin -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap12 \
-s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:12 < /dev/null & sleep 2 && vncviewer 0:12

=

2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c2881000 (size           77f000) map
 CConn:       Utilizzo del formato pixel depth 24 (32bpp) little-endian rgb888
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000072000000 -> 0x00000000c1000000 (size           880000) map
2/0/0 modify_mmio 0x0000000072881000 -> 0x00000000c1881000 (size           77f000) map

Istantanea_2024-07-15_15-27-07.png
 
Instead,adding the -A parameter like this :

Code: 
/usr/sbin/./bhyve-lin -S -c sockets=4,cores=2,threads=1 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap12 \
-s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:12 < /dev/null & sleep 2 && vncviewer 0:12

sometimes is able to boot Windows,but the GPU passthru does not work :

Istantanea_2024-07-15_15-37-07.png
 
The bhyve-win you're using is the same as I have on 14.0-RELEASE-p6, it's the same binary. Their cksums:
Code: 
2173356163 459672 /usr/sbin/bhyve    14.0-RELEASE-p6   
2173356163 459672 bhyve-win

The bhyve-lin is custom built. I suspect not much has changed there either but its entry point is different, everything is moved by a small offset. This makes quick comparison harder, more muscle is needed.

But:
Code: 
[19:55:43] fbsd14f(~)# strings bhyve-lin | grep corvin
/usr/corvin-src-releng-140/usr.sbin/bhyve/acpi_device.c
/usr/corvin-src-releng-140/usr.sbin/bhyve/pci_virtio_scsi.c
/usr/corvin-src-releng-140/usr.sbin/bhyve/rtc.c
..
[19:55:50] fbsd14f(~)#

That would be a start to look at what's changed. It also confirms that both vmm and other stuff you're using are coming from this repo.
 
_martin said:
The bhyve-win you're using is the same as I have on 14.0-RELEASE-p6, it's the same binary. Their cksums:
Code: 
2173356163 459672 /usr/sbin/bhyve    14.0-RELEASE-p6
2173356163 459672 bhyve-win

The bhyve-lin is custom built. I suspect not much has changed there either but its entry point is different, everything is moved by a small offset. This makes quick comparison harder, more muscle is needed.

But:
Code: 
[19:55:43] fbsd14f(~)# strings bhyve-lin | grep corvin
/usr/corvin-src-releng-140/usr.sbin/bhyve/acpi_device.c
/usr/corvin-src-releng-140/usr.sbin/bhyve/pci_virtio_scsi.c
/usr/corvin-src-releng-140/usr.sbin/bhyve/rtc.c
..
[19:55:50] fbsd14f(~)#

That would be a start to look at what's changed. It also confirms that both vmm and other stuff you're using are coming from this repo.
Click to expand...

Since I'm using 14.0,bhyve-win has been built for 14.0. Has been Bileslav to give it to me. At the time it was p6. Correct.

14.0-RELEASE-p6, it's the same binary.
Click to expand...

correct. The only difference is that it has been patched with the KVM string.

The bhyve-lin is custom built. I suspect not much has changed there either but its entry point is different, everything is moved by a small offset. This makes quick comparison harder, more muscle is needed.
Click to expand...

Corvin has patched it to pass the nvidia GPU for the Linux vms. The point of interest here is why it can't be used to pass the GPU to a Windows vm. Can you give a look,please,if bhyve-win has the KVM string inside and bhyve-lin does not ?
The worst hypothesis is that the code that Corvin added inside bhyve-lin (and that's inside his repo),interferes with the passthru of the GPU in Windows. So maybe that code should be rewritten.
 
ziomario said:
correct. The only difference is that it has been patched with the KVM string.
Click to expand...
That's not correct. Only vmm.ko kernel module has bhyve_id string inside.

bhyve-win has the same checksum, i.e. it's identical to, the 14.0-RELEASE-p6 version. Nothing was changed there.
hbyve-lin is a bit diffent. There is information there that points to the location where it was fetched from, its origin. It should be way easier to compare that (though I suck at git, something I want to fix for some time).

Both files are without debug symbols. One entry point (ELF entry point) is different making all offsets different. Binary comparison it's a bit more complicated here.
 
I can help you to understand what's the most relevant part of the code that has been added to bhyve-lin,because I was the one who talked with the nvidia developer,who explained what was missing in the bhyve code to allow the GPU passthru in a Linux vm. You will find it very interesting :

forums.developer.nvidia.com

(EE) no screens found(EE) whyle trying to enable the nvidia driver on Debian 11 emulated with bhyve

This problem is fixed. Hiding the hypervisor worked. Now can we come back where we started ? to the earlier problem ? to bhyve. I will go under FreeBSD and I will try if this working vm runs even there.
forums.developer.nvidia.com forums.developer.nvidia.com

The discussion ended being able to pass the GPU in a Linux vm,but not in a Windows vm. But now,after some years,I see how much close we were to do it.
 
The module I shared, did you try it on both Linux and Windows?

The nvidia thread you shared: well, in 2021 nvidia dev suggested you to fake it with KVM (or hide it, which would be more difficult). So that's where the idea came from.
 
_martin said:
The module I shared, did you try it on both Linux and Windows?
Click to expand...

Where did you share it ? I don't see it.

The nvidia thread you shared: well, in 2021 nvidia dev suggested you to fake it with KVM (or hide it, which would be more difficult). So that's where the idea came from.
Click to expand...

If I remember well,we opted to hide it,not to fake it with KVM. This wasn't the relevant part of the modifications that should have been done. Read well. Bhyve needed a better implementation of the MTRR (Memory Type Range Register) control because the nvidia driver wanted it.

Anyway,actually,if you don't apply the Corvin patches you will be not able to pass your nvidia GPU to a Linux vm. I think that some patches of Corvin haven't been accepted,yet.
 
ziomario said:
Where did you share it ? I don't see it.
Click to expand...
One I shared last time and you tested and said it's working: this one

I read it properly. Hiding string is not replacing it all with 0s. In your post in nvidia forums you actually messed it up because cpuid is returning non-null data.
My hunch is that kvm is hiding the func properly, i.e. returning proper register states as if run on non-virt cpu. But that's beside the point here.
 
_martin said:
One I shared last time and you tested and said it's working. this one
Click to expand...

If you have changed only the string,I should use two different bhyve executables. vmm.ko is not the only player on the playground. Bhyve-lin and bhyve-win are different. So,even if I use the patched vmm.ko,I need to use the first one to pass the GPU on a Linux vm and the second one to pass the GPU to a Windows vm. Your vmm.ko module will not help to use only one bhyve executable,as well as it did not work with the Bileslav module,where the string had already been changed.

Actually I'm using FreeBSD 14.0-p6 with the vmm.ko patched and I tried to passthru the gpu to a Windows vm using bhyve-lin and it didn't work. So this is the proof that bhyve-lin or bhyve-win should be modified in some way.
 
You are not correct. On top of it you are mixing stuff making this really confusing.
ziomario said:
If you have changed only the string,I should use two different bhyve executables.
Click to expand...
No, you should have used patched vmm with GENERIC, official userland bhyve executables.

I already told you bhyve-win is just bhyve from official release. It's the bhyve-lin that you use is non-standard. Question is what else did you overwrite with something else.

Point of patching vmm.ko was to see if windows nvidia driver under bhyve was fooled by just a simple trick of renaming the hypervisor's id to KVM. You should not have used any 3rd party, unknown to us userland binaries to achieve that. If you did test is inconclusive.

Point of the test is to take vmm.ko with the patched string and try to passthrough and install drivers uder Windows and Linux using official bhyve executables.

Btw. patches in question are probably these.
 
_martin said:
No, you should have used patched vmm with GENERIC, official userland bhyve executables.
Click to expand...

Generic,official userland bhyve executable = bhyve-win in my case (I have renamed it).
So,vmm.ko patched + bhyve-win = ok IF I want to pass the GPU inside a Windows vm.
If I want to pass the GPU inside a Linux vm I should use : vmm.ko (patched or not it makes no difference) + bhyve-lin

Take in consideration that if I use the not patched version of vmm.ko,I can't pass the GPU to Windows (but I can do it for Linux),so I must use the patched version (and two different bhyve executables) to pass the GPU one time on Linux and one time on Windows.

Today I showed what happens if I pass the GPU using bhyve-lin inside a Windows vm : crash and / or GPU can't be passed.

I repeat : the goal is to use ONE only bhyve executable (I think it is bhyve-lin,because it is already has the patches for passing the GPU to a Linux vm) to pass the GPU to both the OS.

In any case,at the moment everything is working for me,even if I should use two different bhyve executables.
 
You must log in or register to reply here.
Back
Top