Solved Trying to understand why only some kind of vmm.ko module allows to passthru my nvidia GPU to a WIndows 11 vm

x86/x86/identcpu.c:1351

Code: 
static struct {
        const char      *vm_cpuid;
        int             vm_guest;
} vm_cpuids[] = {
        { "XenVMMXenVMM",       VM_GUEST_XEN },         /* XEN */
        { "Microsoft Hv",       VM_GUEST_HV },          /* Microsoft Hyper-V */
        { "VMwareVMware",       VM_GUEST_VMWARE },      /* VMware VM */
        { "KVMKVMKVM",          VM_GUEST_KVM },         /* KVM */
        { "bhyve bhyve ",       VM_GUEST_BHYVE },       /* bhyve */
        { "VBoxVBoxVBox",       VM_GUEST_VBOX },        /* VirtualBox */
};

ETA: never mind, that is guest code.
 
cracauer@ said:
x86/x86/identcpu.c:1351

Code: 
static struct {
        const char      *vm_cpuid;
        int             vm_guest;
} vm_cpuids[] = {
        { "XenVMMXenVMM",       VM_GUEST_XEN },         /* XEN */
        { "Microsoft Hv",       VM_GUEST_HV },          /* Microsoft Hyper-V */
        { "VMwareVMware",       VM_GUEST_VMWARE },      /* VMware VM */
        { "KVMKVMKVM",          VM_GUEST_KVM },         /* KVM */
        { "bhyve bhyve ",       VM_GUEST_BHYVE },       /* bhyve */
        { "VBoxVBoxVBox",       VM_GUEST_VBOX },        /* VirtualBox */
};

ETA: never mind, that is guest code.
Click to expand...

what's this mean ?
 
ziomario said:
I have been able to passhtru my GeForce RTX 2080 ti to a bhyve virtual machine running with Windows,overcoming/fixing the errors that I've got for years,errors 43 and 12,in a stable and permanent way.
Click to expand...
Who reported this error: host (FreeBSD) or guest (Windows)?

If guest I wonder if nvidia driver includes code to deal with qemu's kvm and hence is able to attach driver in Windows because it recognizes the /supported/ virtualization.
 
_martin said:
Who reported this error: host (FreeBSD) or guest (Windows)?

If guest I wonder if nvidia driver includes code to deal with qemu's kvm and hence is able to attach driver in Windows because it recognizes the /supported/ virtualization.
Click to expand...

Common errors on Windows guest : error 12 or 43.
 
jbo@ said:
I am experiencing issues with PCI passthru but I can't say whether it's a FreeBSD problem: https://forums.freebsd.org/threads/bhyve-vm-stuck-when-passthru-enabled.92854/#post-648828

After updating to a recent stable/14 a few days ago I tested again and I'm still observing the same behavior.

The current assumption is that the problem is related specifically to Windows 11 as per bileslav suggestion. Whether it's a bhyve issue, an EDK issue or an "issue" inside of Windows itself is yet to be investigated.
Click to expand...

Can you try to set the string ?
 
I think that if it's possible to replace the string manually without recompiling the source code is better,some time will be saved. I did this using HxD when I changed the string inside the nvidia driver file,but I wonder if the process can be automatized,something like we create a script and when we click there it changes the string for every vmm.ko that has already been compiled...someone wants to try ?
 
Yes, it can be. Similar to cracks one applies to binary. Technically we are not replacing string but rather patching asm instructions (bcopy() got optimized and string in .data /bhyve_id/ is opted out).
Such thing is not needed when you have sources as you can easily just change it and recompile given module.

As part of fun and 'hacking competition' I'd create a task to change it to sysctl variable one can set on demand (beware of race conditions). Of course not something that would be officially in repos, really just as a fun hack project.

Officially I'd contact NVIDIA and asked them about this. Most likely driver is trying to detect hypervisor platform (cpuid func 0x40000000) and rejects bhyve. Again, just a hunch.
 
I've asked to my friend how to patch the binary without recompiling it from the source code and he replied like this :

The right way to do that is to patch bhyve code and recompile it. Another possible solution is to use hypervisor hiding mode, so far I remember that helped with KVM.

But if you really wants to patch the binary you can use tools like sfk. On windows the command line should be like this:
sfk.exe replace -bin /bb6268797641bf6520626841b879766520/bb4b564d4b41bf564d4b5641b84d000000/ -dir "C:\vmmko_dir" -file vmm.ko -yes

This will find the byte pattern of the original code and replace it with the new one. The pattern is unique, I dont think there will be other such patterns in the code, but if a newer version will use a different pattern (write to other registers, use other write commands, etc), the pattern will change and this command line will not find it.
Another possible problem - module integrity checks. If vmm.ko has checksums or signatures these must be updated.
Click to expand...

So,some questions :

1) I don't want to use sfk for Windows,but a similar tool under FreeBSD :)
2) vmm.ko has checksums or signatures ? If yes,how can they be updated ?

thanks.
 
Focus, man, focus.

Verify that plain 14.1 as built from source by you doesn't work. Then patch the source code of vmm.ko and verify that does work.

You don't have to recompile the world to compile vmm.ko. You can compile the module separately, or just the kernel.
 
Man,I have more focus than you,since I don't want neither compile vmm.ko,but I want to reuse already compiled old and future vmm.ko modules :D
 
cracauer@ said:
Verify that plain 14.1 as built from source by you doesn't work. Then patch the source code of vmm.ko and verify that does work.
Click to expand...

I've upgraded 14.0 to 14.1 and I have changed the string inside x86.c ; it worked. vmm.ko for 14.1 allows to passthru the nvidia gpu correctly. Now I want to patch the module like my friend suggested. But before I want to find an alternative tool for sfk. I want to do it directly in FreeBSD.
 
It's insane to use binary patching when source code is available.

But on the bright side a binary patcher is a nice tool to master. Comes in handy later. Let us know what you end up liking.
 
ziomario said:
Now I want to patch the module like my friend suggested.
Click to expand...
The actual patch itself is easy. But you need to know what to patch it with and where. You can't assume blob of data e.g. beef00cafe will work on the 14.2 as compiler could change things (as it was in 14.0 to 14.1 where compiled code used different register, hence different code).

As mentioned this few times here, other mentioned here too: it can't get easier than having source code and way of compiling it.

AFAIK secureboot is not working on FreeBSD, no module signing and verification is in place. At least not yet. Hence you can modify and load anything (as you did when you loaded my modified module).

Look at bvi(1), my weapon of choice when it comes to binary patching. If I need to write a reusable crack then I go with C and do the: verify, save, modify.
 
In case you didn't get how easy it is to compile the kernel module:

a) change what was mentioned above
b)
Code: 
cd /usr/src/sys/modules/vmm
make
You have your module ready in /usr/obj/usr/src/amd64.amd64/sys/modules/vmm/vmm.ko. Copy it where you need (make a copy of an original one and copy it over there once you test all is ok).
Way easier than trying to grasp binary patching.


I grabbed a :beer: and let myself have some fun.

To detect what hypervisor is system runing under (makes sense under VM):
Code: 
.section .text
    .globl _start

_start:
    movl $4, %eax
    movl $1, %edi
    leaq (msg), %rsi
    movl $(buf-msg-1), %edx
    syscall

    movl $0x40000000, %eax
    cpuid

    leaq (buf), %rdi
    movl %ebx, (%rdi)
    movl %ecx, 4(%rdi)
    movl %edx, 8(%rdi)
    movb $0xa, 12(%rdi)

    movl $13, %edx
    movq %rdi, %rsi
    movl $1, %edi
    movl $4, %eax
    syscall

    movl $1, %eax
    xorl %edi, %edi
    syscall


.section .data
    msg:    .asciz    "detected hypervisor: "
    buf:    .quad    0,0

Compile with (using as from gcc):
Code: 
    as -o info.o info.S
    ld -o info info.o

Let's see what is does under bhyve:
Code: 
vm01(~)# ./info
detected hypervisor: bhyve bhyve
vm01(~)#

Now little "tweak" on FreeBSD host:
Code: 
[22:10:58] fbsd14f(~)# sysctl hw.vmm.bhyve_id
hw.vmm.bhyve_id: bhyve bhyve
[22:11:01] fbsd14f(~)#

[22:11:02] fbsd14f(~)# sysctl -d hw.vmm.bhyve_id
hw.vmm.bhyve_id: martin having some beer fun with bhyve_id
[22:11:02] fbsd14f(~)#

[22:11:03] fbsd14f(~)# sysctl hw.vmm.bhyve_id="KVMKVMKVM"
hw.vmm.bhyve_id: bhyve bhyve  -> KVMKVMKVM
[22:11:08] fbsd14f(~)#

What does our VM see now ?
Code: 
vm01(~)# ./info
detected hypervisor: KVMKVMKVM
vm01(~)#
 
I'd say the only proper course of action is for ziomario to verify, with 100% certainty, that this is the only thing keeping Nvidia GPU passthru from working and then opening a corresponding forum post / support ticked over at the Nvidia forum while everybody else will pray for Nvidia to update their driver accordingly.

Nvidia, if you're reading this: Despite all the crap you're pulling lately, I'm still only buying Nvidia GPUs because you provide native FreeBSD drivers. You rock!
 
Why the string is not present on the vmm.ko source code of FreeBSD ? It seems relevant. It enables a very nice and useful function. It fixes the error 43 given by Windows maybe because it is not able to recognizes the nvidia drivers,because inside their code there is a protection check that refuse the installation if it does not find the KVM hypervisor. More than asking to change the check inside the nvidia driver,that it will never happen,I would ask why no one proposed to change by default string bhyve bhyve with KVMKVMKVM for every stock image of FreeBSD that comes out.
 
ziomario said:
Why the string is not present on the vmm.ko source code of FreeBSD ? It seems relevant.
Click to expand...
It is present in vmm.ko. But it reads "bhyve bhyve " for obvious reasons (because the hypervisor is bhyve).
Setting it to "KVMKVMKVM" is really just a dirty workaround.

ziomario said:
I would ask why no one proposed to change by default string bhyve bhyve with KVMKVMKVM for every stock image of FreeBSD that comes out.
Click to expand...
That is not an option. That string identifies the hypervisor. FreeBSD's hyvervisor is bhyve, not KVM.
The correct fix is to ask Nvidia nicely to update their driver to work with that CPUID.
Your proposal is similar to asking all Intel CPUs to identify themselves as AMD CPUs - that makes no sense and will never happen (I hope).
 
ziomario said:
Why the string is not present on the vmm.ko source code of FreeBSD ?
Click to expand...
I presented the exact location and explanation more than once in this thread. Also with the explanation why it's not in the .data section of the module.

You can't ask FreeBSD to change their hypervisor id just because some windows driver is ignoring something. This is hack, dirty one at that. Be happy that it works because that is one lucky quirk on nvidia side too.
 
Correct me if I'm wrong,but the code of the nvidia driver for FreeBSD is the same code of the Linux driver. Basically nvidia is not spending money to develop a native driver,they spend 0 because they are reusing already written code. Let's say what's the truth : nvidia does not care about FreeBSD. Only occasionally they can provide some focused fix for us. It means that maybe we should be more practical. So,the focus could become to enable a function regardless of formalisms and claims. Some parts of the FreeBSD code needs parts of Linux code. Maybe when it will become totally independent from Linux and it will have acquired much more popularity (mainly on the Desktop area,where nvidia cares a lot), nvidia will start to consider FreeBSD a separate OS from Linux.
 
You must log in or register to reply here.
Back
Top