Solved Trying to understand why only some kind of vmm.ko module allows to passthru my nvidia GPU to a WIndows 11 vm

You must understand we don't know your setup nor what you did change on the system. I can't test it myself, I don't have HW for it. For other readers/googlers it will be important to know they can fool (for now) windows drivers.

If you said and confirmed you didn't change any userland tools and was able to pass through with modified vmm in windows then there's partial success.
Linux patches are patching more stuff (I pasted the link above what is most likely applied to your bhyve-lin). Interestingly those patches of vmm there are not included in your vmm you initially shared.
But if that works and those patches are those I shared above I guess they will get in upstream when they are mature enough.
 
The point is that the final solution can't be to use two different bhyve executables. Doing like this sounds too much like an hack,like a temp solution. Also note that Bileslav (yeah,again him),told me that on 14.0-p6 he was able to use one only bhyve executable for both the OS. Oh my god,I'd forgotten that I have the file that he gave to me and that it should work for both the OS but it doesn't (for me). Do you want that I share it ? For sure you want !

bhyve-lin+win

drive.google.com drive.google.com

So,since I believe in him,I think that this can be done. But only he knows how. In the meantime you could look inside of it.
 
For me, it wasn't necessary and still isn't necessary to fool the driver with these KVMs, passthru works with bhyve bhyve ; and I'm not the only one like this. So it's all a little weird. Again, I'm talking only about Windows 10.

_martin said:
Btw. patches in question are probably these.
Click to expand...
https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip

Those are fixes to allow NVIDIA GPU passthru to Linux VMs. For me, the patches don't break the passhtru to Windows. I never tried to compile that on 14.1, though.

ziomario said:
Also note that Bileslav (yeah,again him),told me that on 14.0-p6 he was able to use one only bhyve executable for both the OS. Oh my god,I'd forgotten that I have the file that he gave to me and that it should work for both the OS but it doesn't (for me). Do you want that I share it ? For sure you want !
Click to expand...
It was compiled with CPUTYPE?=znver3, so it's useless for you, and I told you this. Besides the patches linked above, there's nothing special about that binary. It would be better if you provided the information like this, instead of sending the file.
 
Ha. Yes, indeed, exactly here the substitution was made, which you all here had to heroically discover. That is, it's not from me (I'm sorry to say this again). The weirdest thing about all this is that ziomario assures that it's with these patches the passthru to Windows stops working for him, so he needs two different /usr/sbin/bhyve; but at the same time, he presents that patched vmm.ko as a working one.
 
bileslav said:
For me, it wasn't necessary and still isn't necessary to fool the driver with these KVMs
Click to expand...
Really just for my curiosity, what version of guest driver are you using?

bileslav said:
Those are fixes to allow NVIDIA GPU passthru to Linux VMs
Click to expand...
Right, it was shared in the context of trying to figure out why ziomario has bhyve-lin and what is the probable origin of those. There's also the bhyve_id mods in there, among other things (hiding hypervisor etc.). And yet his module he shared here doesn't have it. That's why I mentioned "probably" too.

The nvidia forum post show that the idea of fooling the hypervisor came from there around 2021. Then the other blog post shared the similar approach.

At this point this thread is way too complicaited and includes way too many /mis/information. Hopefuly others can get something out of it.

I may look at ziomario's last binary he sent but I'd need to RE it in IDA, this is too much for static analysis. But I don't think it would be usefull to waste my time like this.
 
bileslav said:
Ha. Yes, indeed, exactly here the substitution was made, which you all here had to heroically discover. That is, it's not from me (I'm sorry to say this again). The weirdest thing about all this is that ziomario assures that it's with these patches the passthru to Windows stops working for him, so he needs two different /usr/sbin/bhyve; but at the same time, he presents that patched vmm.ko as a working one.
Click to expand...

That's wrong. Putting the KVMKVMKVM string inside the vmm.ko the passthru to Windows starts working,but only using bhyve-win,that I've talked about above,not using bhyve-lin,that's bhyve patched by Corvin. So,I'm using one only vmm.ko (the one with the string inside) and two bhyve executables). That's what I should do to have the passthru working everywhere. Yes,it's more complicated than what Bileslav does usually to. But is that my fault ? I accurately test and retest every experiment I do.
 
bileslav said:
It was compiled with CPUTYPE?=znver3, so it's useless for you, and I told you this. Besides the patches linked above, there's nothing special about that binary. It would be better if you provided the information like this, instead of sending the file.
Click to expand...

ehy,I forgot that. But giving that file to _martin he will find it by himself.
 
ziomario said:
Generic,official userland bhyve executable = bhyve-win in my case (I have renamed it).
So,vmm.ko patched + bhyve-win = ok IF I want to pass the GPU inside a Windows vm.
If I want to pass the GPU inside a Linux vm I should use : vmm.ko (patched or not it makes no difference) + bhyve-lin
Click to expand...
Just to remove every possible doubt (and I'm preventively sorry if this has been addressed already and I missed it), but did you try GPU passthru with a Linux VM using bhyve-win?
 
ziomario said:
GPU passthru to a Linux VM using bhyve-win can't work because bhyve-win is not patched with the Corvin's patches.
Click to expand...
Yeah, I forgot that you are using 14.0 and not 14.1; I hoped that Corvin's patches had been merged in 14.1 (as supposed by bileslav) and you could try this way to use the stock bhyve binary with just vmm.ko patched.
 
Maxnix said:
Yeah, I forgot that you are using 14.0 and not 14.1; I hoped that Corvin's patches had been merged in 14.1 (as supposed by bileslav) and you could try this way to use the stock bhyve binary with just vmm.ko patched.
Click to expand...

I've upgraded 14.0 (zfs system) to 14.1 ; Corvin's patches haven't been merged to 14.1,but this is not so important,because on the new 14.1 system I can reuse the bhyve-win and bhyve-lin executables that I use on 14.0 ; regarding vmm.ko,I've recompiled it by changing the string. So,GPU passthru still work for Linux and Windows vms even on the 14.1.
 
_martin said:
Really just for my curiosity, what version of guest driver are you using?
Click to expand...
It starts working right after auto-installation of the driver via Windows Update. Then it continues to work after installing GeForce Experience and updating it all from there. I can't tell the exact numbers because I once again made a mess on my computer.

_martin said:
There's also the bhyve_id mods in there, among other things (hiding hypervisor etc.). And yet his module he shared here doesn't have it. That's why I mentioned "probably" too.
Click to expand...
Hmm, yes, I see 3 commits in x86.c. So if you're sure that the vmm.ko he shared doesn't have that, it clearly indicates that the module is of a different origin.

_martin said:
But I don't think it would be usefull to waste my time like this.
Click to expand...
As I said, besides Corvin's patches, there's nothing special about the binary he sent, so yes.

ziomario said:
That's wrong.
Click to expand...
I don't see anything wrong with that. Corvin's patches affect not only the executable file, but also the kernel module, and you have to use the kernel module (patched with KVMKVMKVM), but not the executable, to make passthru work with Windows. However, if your vmm.ko doesn't come from their repo, my message becomes a little meaningless.

Maxnix said:
I hoped that Corvin's patches had been merged in 14.1 (as supposed by bileslav)
Click to expand...
ziomario said:
Corvin's patches haven't been merged to 14.1
Click to expand...
Yes, unfortunately, that hypothesis wasn't confirmed.
 
bileslav said:
I don't see anything wrong with that. Corvin's patches affect not only the executable file, but also the kernel module, and you have to use the kernel module (patched with KVMKVMKVM), but not the executable, to make passthru work with Windows. However, if your vmm.ko doesn't come from their repo, my message becomes a little meaningless.
Click to expand...

For the passhthru of Windows I use the bhyve executable without the Corvin patches. I renamed it as bhyve-win. It does not need of them,otherwise it does not work. What should still be done is to use one only bhyve executable for the passthru of Windows and Linux. This is valid for me,but not for you,since you said that you can use one only bhyve executable for both the OS. This is the point that still need to be explored. I don't know how many users have my same situation. A nice interesting experiment could be to try different versions of the bhyve executable,to see if the GPU passthru for Windows works with every one of them.
 
bileslav said:
Yes, unfortunately, that hypothesis wasn't confirmed.
Click to expand...

Why unfortunately ? I'm using 14.1 and I can use the GPU passthru in Linux and Windows. You don't need to wait that Corvin rebase his code for the 14.1,even if it's not a bad idea if he does it.
 
It depends.
If the actual GPU and/or some important components do not 100% match as yours (includes chip/PCB revisions, not only models), the hardware could work differently even on same driver and OS. So statement "unfortunately" looks quite reasonable here.
 
bileslav said:
Besides, I believe it's just wrong to use patches made for another branch, even if it seems to work in this case.
Click to expand...

Sure,but what should I do if Corvin does not do it ? Should I renounce ? I try to solve the problem(s) doing the best as I can. I want / need to have the functionality. For example I've just "burnt" the subtitles to a movie and I made it using handbrake which works in Windows. It is a good tool that made the job very quickly. Windows sometimes is useful. Do this task in FreeBSD would have been a challenge.
 
Interesting to know if NapoleonWils0n is interested to explore how can be done that particular task using FreeBSD only ;D. That's for sure another argument,so the answer should not be placed here.
 
ziomario said:
For example I've just "burnt" the subtitles to a movie and I made it using handbrake which works in Windows.
Click to expand...

ziomario said:
Interesting to know if NapoleonWils0n is interested to explore how can be done that particular task using FreeBSD only ;D. That's for sure another argument,so the answer should not be placed here.
Click to expand...

That's a simple one-liner with ffmpeg: ffmpeg -i video.mp4 -vf subtitles=subtitles.srt output-video-with-subtitles.mp4

There's no need to touch that redmond bloatware for such simple tasks (which are overly tedious on that advertisement-platform anyways...)


edit:
NapoleonWils0n that's a nice collection of scripts. Repo starred and bookmarked, Thanks!
 
ziomario said:
That's good. I didn't think that it could have existed.
Click to expand...
I learned that is always worth checking for the availability of a given software for a specific OS, or if not available for a (valid) alternative. This is especially true when open source softwares are involved. They are usually really multi-platform. :)
 
You must log in or register to reply here.
Back
Top