Solved Trying to understand why only some kind of vmm.ko module allows to passthru my nvidia GPU to a WIndows 11 vm

cracauer@ :

Do you want to give a look inside the vmm.ko that should work for FreeBSD 14.1 ? I'm curious to see if you see the same pattern there. The module comes again from the Bileslav PC. Actually it does not work,but I would like to appurate if it really does not work,because maybe I made some mistake. If you find the same pattern that you have found on the module which works for the 14.0,maybe even the module for 14.1 works and then we can be almost sure that Bileslav is using some special flag or a different compiler or something that can repeatedly help.

vmm.ko module produced by upgrading 14.0 to 14.1 using freebsd-update :

vmm.ko

drive.google.com drive.google.com
 
ziomario said:
cracauer@ :

Do you want to give a look inside the vmm.ko that should work for FreeBSD 14.1 ? I'm curious to see if you see the same pattern there. The module comes again from the Bileslav PC. Actually it does not work,but I would like to appurate if it really does not work,because maybe I made some mistake. If you find the same pattern that you have found on the module which works for the 14.0,maybe even the module for 14.1 works and then we can be almost sure that Bileslav is using some special flag or a different compiler or something that can repeatedly help.

vmm.ko module produced by upgrading 14.0 to 14.1 using freebsd-update :

vmm.ko

drive.google.com drive.google.com
Click to expand...

Sure.
 
ziomario said:
The compiler differences are on the Bileslav PC or in the compiler used by the developer that created or compiled vmm.ko ?
Click to expand...
ziomario said:
bileslav : what do you think now ?
Click to expand...
It seems you continue to think that I did not take my vmm.ko from the official FreeBSD distribution, although you found a binary-identical vmm.ko there by yourself. I'm tired of proving the obvious fact that you simply ignore.
 
ziomario said:
If you find the same pattern that you have found on the module which works for the 14.0,maybe even the module for 14.1 works and then we can be almost sure that Bileslav is using some special flag or a different compiler or something that can repeatedly help.
Click to expand...
Oh my goodness. This wall is impenetrable. Would you like me to send you some another file from the FreeBSD base (which everyone can get from there with their own hands, including you), so that you can then try to convince people of its uniqueness? Just kidding, I won't do this again.

Sorry, this is driving me crazy.
 
PMc said:
So why don't you just try it out, change the origin tag, recompile and see if that makes the difference?
Click to expand...

I have already upgraded 14.0 to 14.1 using freebsd-update and I've used the vmm.ko provided by Bileslav. It has been accepted,but the GPU has not been passed correctly. Do you suggest to follow a different strategy ?
 
bileslav said:
Oh my goodness. This wall is impenetrable. Would you like me to send you some another file from the FreeBSD base (which everyone can get from there with their own hands, including you), so that you can then try to convince people of its uniqueness? Just kidding, I won't do this again.

Sorry, this is driving me crazy.
Click to expand...
Well, if we believe in cracauer@'s disassembly, then somebody has changed the origin tag. Compilers usually do not modify static strings on their own behalf.
 
ziomario said:
I have already upgraded 14.0 to 14.1 using freebsd-update and I've used the vmm.ko provided by Bileslav. It has been accepted,but the GPU has not been passed correctly. Do you suggest to follow a different strategy ?
Click to expand...
I have no idea what You are doing. And I never used Windows.
 
PMc said:
bileslav said:
It seems you continue to think that I did not take my vmm.ko from the official FreeBSD distribution, although you found a binary-identical vmm.ko there by yourself. I'm tired of proving the obvious fact that you simply ignore.
Click to expand...

Because the vmm.ko provided by the official FreeBSD distribution does not allow my GPU to be passed. Your file works. And the files aren't identical. cracauer@ shows some differences,not in the code,but it seems in the compilation. Anyway,it is not yet clear.
Click to expand...
 
bileslav said:
Nice, but how am I related to that? Please, read this message. In that message, I'm talking about this. I invite everyone here to read that.
Click to expand...
Sorry, but this is none of my concern. I don't care about who sent what to whom about where and whence - because that is not even resolveable from my location in any case.
But then cracauer@ disassembled two images (again, I have no idea where these came from) and presented the difference - and I believe him doing this correctly - and I find the difference interesting. And that is all.
 
PMc said:
Sorry, but this is none of my concern. I don't care about who sent what to whom about where and whence - because that is not even resolveable from my location in any case.
But then cracauer@ disassembled two images (again, I have no idea where these came from) and presented the difference - and I believe him doing this correctly - and I find the difference interesting. And that is all.
Click to expand...

For me it's not a problem if Cracauer or someone else wants to choose a different vmm.ko module. I will use the one that's working and I can show, by recording a video, what happens,what works and what does not. Where it come from is relevant because if I find an official vmm.ko which enable the passthru of my GPU,I have fixed the problem since I can find the way to upgrade it to later FreeBSD versions.
 
bileslav : there still could be a small chance that I'm not using the vmm.ko that you have given to me. Maybe for a mistake I'm using a vmm.ko that I've got in a different place. If this is important for you,let's make it. If you want to send me again that file AND it will work,there will be 0 chances thats not your file the right one.
 
PMc said:
I thought that much is obvious: x86_emulate_cpuid+0x73 equates to this.
Click to expand...
Yes, indirectly. It's this part of switch statement.

Working part:
Code: 
    b3b7: 41 b9 00 00 00 40             mov     r9d, 0x40000000
    b3bd: bb 4b 56 4d 4b                mov     ebx, 0x4b4d564b
    b3c2: 41 bf 56 4d 4b 56             mov     r15d, 0x564b4d56
    b3c8: 41 b8 4d 00 00 00             mov     r8d, 0x4d
    b3ce: e9 61 05 00 00                jmp     0xb934 <x86_emulate_cpuid+0x5d4>
...
 b934: 44 89 c8                      mov     eax, r9d
    b937: 48 89 07                      mov     qword ptr [rdi], rax
    b93a: 89 d8                         mov     eax, ebx
    b93c: 48 8b 4d 90                   mov     rcx, qword ptr [rbp - 0x70]
    b940: 48 89 01                      mov     qword ptr [rcx], rax
    b943: 44 89 f8                      mov     eax, r15d
    b946: 48 89 06                      mov     qword ptr [rsi], rax
    b949: 44 89 c0                      mov     eax, r8d
    b94c: 48 8b 4d 88                   mov     rcx, qword ptr [rbp - 0x78]
    b950: 48 89 01                      mov     qword ptr [rcx], rax
    b953: b8 01 00 00 00                mov     eax, 0x1
    b958: 48 83 c4 58                   add     rsp, 0x58
    b95c: 5b                            pop     rbx
    b95d: 41 5c                         pop     r12
    b95f: 41 5d                         pop     r13
    b961: 41 5e                         pop     r14
    b963: 41 5f                         pop     r15
    b965: 5d                            pop     rbp
    b966: c3                            ret
So set regs[] and jump to cleanup and ret.

It seems that in working module "KVMKVM" is used, non-working one has "bhyve bhyve" bhyve_id. Quick check on my 13.x and 14.x say bhyve_id was always "bhyve bhyve".
As you mentioned, easy to trick with hex editor to see it this trick works.
 
_martin said:
It seems that in working module "KVMKVM" is used, non-working one has "bhyve bhyve" bhyve_id. Quick check on my 13.x and 14.x say bhyve_id was always "bhyve bhyve".
Click to expand...

So what does that mean?

My own guests running within bhyve show this:
Hypervisor: Origin = "bhyve bhyve "

My rented cloud instances, however, show this:
CPU: QEMU Virtual CPU version 2.5+ (2994.54-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x663 Family=0x6 Model=0x6 Stepping=3 ... Hypervisor: Origin = "KVMKVMKVM"
These CPU designators are quite bogus (if such a thing would exist in reality, it wouldn't support 64bit), and the riddle here is, which CPUTYPE to compile for. But that KVMKVMKVM seems to belong to QEMU.
 
ziomario What can be interesting to you to test this on 14.1. Right now my VM is:
Code: 
FreeBSD fbsd14f 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
I binary patched the original module to mimic your working module:

Code: 
2c2
< /boot/kernel/vmm.ko:    file format elf64-x86-64
---
> vmm.ko:    file format elf64-x86-64
12314,12316c12314,12316
<     b5fd: bb 62 68 79 76                   movl    $0x76796862, %ebx       # imm = 0x76796862
<     b602: 41 bf 65 20 62 68                movl    $0x68622065, %r15d      # imm = 0x68622065
<     b608: 41 be 79 76 65 20                movl    $0x20657679, %r14d      # imm = 0x20657679
---
>     b5fd: bb 4b 56 4d 4b                   movl    $0x4b4d564b, %ebx       # imm = 0x4B4D564B
>     b602: 41 bf 56 4d 4b 56                movl    $0x564b4d56, %r15d      # imm = 0x564B4D56
>     b608: 41 b8 4d 00 00 00                movl    $0x4d, %r8d
I attached it as .txt to bypass filter here, please rename it to vmm-patched.ko or something like that.

Binary diff can be done by:
Code: 
objdump -d /boot/kernel/vmm.ko > orig
objdump -d /path/to/downloaded/module/vmm.ko > patched
diff -w orig patched

I guess it's worth trying for you to test if you can actually fool the pass-through like this.
 

Attachments

  • vmm.ko.txt
    500.7 KB · Views: 14
PMc said:
So what does that mean?
Click to expand...
It's just the vCPU ID as can be seen also here.

It's one of the flag one can detect what hypervisor it's being run on. Some other part of the code can, and some do, make some assumptions based on it.

From what was presented here it seems vmm module had ID changed to be pretending to be something different (i.e. vcpu is not one from bhyve) in order to fool the passthrough. That's just rough hunch though based on very little we have going on here. And it would be a cheap lucky trick too.
 
_martin said:
ziomario What can be interesting to you to test this on 14.1. Right now my VM is:
Code: 
FreeBSD fbsd14f 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
I binary patched the original module to mimic your working module:

Code: 
2c2
< /boot/kernel/vmm.ko:    file format elf64-x86-64
---
> vmm.ko:    file format elf64-x86-64
12314,12316c12314,12316
<     b5fd: bb 62 68 79 76                   movl    $0x76796862, %ebx       # imm = 0x76796862
<     b602: 41 bf 65 20 62 68                movl    $0x68622065, %r15d      # imm = 0x68622065
<     b608: 41 be 79 76 65 20                movl    $0x20657679, %r14d      # imm = 0x20657679
---
>     b5fd: bb 4b 56 4d 4b                   movl    $0x4b4d564b, %ebx       # imm = 0x4B4D564B
>     b602: 41 bf 56 4d 4b 56                movl    $0x564b4d56, %r15d      # imm = 0x564B4D56
>     b608: 41 b8 4d 00 00 00                movl    $0x4d, %r8d
I attached it as .txt to bypass filter here, please rename it to vmm-patched.ko or something like that.

Binary diff can be done by:
Code: 
objdump -d /boot/kernel/vmm.ko > orig
objdump -d /path/to/downloaded/module/vmm.ko > patched
diff -w orig patched

I guess it's worth trying for you to test if you can actually fool the pass-through like this.
Click to expand...

Thanks. But sorry,it didn't work. The nvidia drivers haven't been fooled. Did you forget something ? I think the road is the right one. Which starting vmm.ko module have you got ?
 
Ok, thanks for trying, I was curious myself.

When I booted bhyve VM with my modified module to test this I saw:

Code: 
Hypervisor: Origin = "KVMKVMKV\M^PLv\^E"[

Basically string that is copied over to those regs is not properly done. But disassembly of your working module to this one is the same (i.e. same part of the code was changed).

Just to verify can you boot FreeBSD under byve with the module where passthrough is working and check for this string in dmesg ?
 
Istantanea_2024-07-12_01-57-18.png
 
Seems to be the thread to lock over the weekend so no one runs off the rails. Is this the case? You tell me.
 
ziomario said:
can you explain this ? what do you do there ?
Click to expand...

Briefly:
  1. use FreeBSD-14.0-RELEASE-amd64-dvd1.iso.xz to install 14.0-RELEASE
  2. start the installed system
  3. refrain from using freebsd-update
  4. initialise pkgbase
  5. restart the OS.
Result (output from uname -mvKU):

FreeBSD 14.0-RELEASE-p8 releng/14.0-n265421-5e23806790ef GENERIC amd64 1400097 1400097
  • neither an incremental kernel build number, nor the name of the computer that performed the build
  • patch level 8
  • 5e23806 is the uppermost commit at the FreshBSD page for the releng/14.0 branch.

Later today, probably this evening, I'll do something similar with 14.1. Aiming to:
  1. use FreeBSD-14.1-RELEASE-amd64-dvd1.iso.xz to install 14.1-RELEASE
  2. refrain from exiting the installer
  3. initialise pkgbase
  4. exit
  5. start the installed system – 14.1-RELEASE-p2 from the outset.
 
You must log in or register to reply here.
Back
Top