bhyve Is there a method to understand inside which FreeBSD branch does my vmm.ko file come from ?

ziomario said:
This is a message shown on the FreeBSD ML some days ago :
Click to expand...
Who said it's possible to use loadable kernel drivers between different version of the FreeBSD kernel should proof it.

Point it out in https://docs.freebsd.org/en/books/arch-handbook/book, or proof it on a running system.

If it's possible I stand corrected, but it doesn't look like it's possible.

kldload-vmm.ko.png


Except vmm_140.ko (copied from a installed system) all other modules are build from local source ( src/sys/modules/vmm/ ), after checkout of respective branch.
 
The error message in /var/log/messages is correct but the error message "kldload: can't load ./vmm.ko: module already loaded or in kernel" printed by kldload should be fixed.
 
cracauer@ said:
Do you know what a patch or a diff is?
Click to expand...

I think I can apply a patch following these explanations :

forums.freebsd.org

Solved - Applying an unofficial patch on a port

Hi, Noob question maybe but I am kind of desperate here. I can't figure out how to apply a custom patch on a port. I only have basic knowledge about ports. My company is asking me to make security/teleport version 8.x.x or more work on FreeBSD. The current version present in ports/pkg (4.3.9)...
forums.freebsd.org forums.freebsd.org

but I'm not able to understand where I can find the patch.
 
ziomario said:
I think I can apply a patch following these explanation :

forums.freebsd.org

Solved - Applying an unofficial patch on a port

Hi, Noob question maybe but I am kind of desperate here. I can't figure out how to apply a custom patch on a port. I only have basic knowledge about ports. My company is asking me to make security/teleport version 8.x.x or more work on FreeBSD. The current version present in ports/pkg (4.3.9)...
forums.freebsd.org forums.freebsd.org

but I'm not able to understand where I can find the patch.
Click to expand...

Instead of accepting a binary kernel module you should have asked the author for a patch.
 
T-Daemon said:
Who said it's possible to use loadable kernel drivers between different version of the FreeBSD kernel should proof it.

Point it out in https://docs.freebsd.org/en/books/arch-handbook/book, or proof it on a running system.

If it's possible I stand corrected, but it doesn't look like it's possible.

View attachment 19434

Except vmm_140.ko (copied from a installed system) all other modules are build from local source ( src/sys/modules/vmm/ ), after checkout of respective branch.
Click to expand...

you are right :

==> mousepad /mnt/ada2p2/var/log/messages

Jul 5 12:36:39 marietto kernel: KLD vmm.ko: depends on kernel - not available or version mismatch
 
cracauer@ said:
Instead of accepting a binary kernel module you should have asked the author for a patch.
Click to expand...

The author does not know anything about what patches have been applied. He used the vmm.ko provided by FreeBSD 14.0 and he gave it to me and it worked for me. But now I have upgraded 14.0 to 14.1 and that module does not work anymore. In the meantime he upgraded 14.0 to 14.1 also and I've asked to him the vmm.ko module for 14.1 and the boot directory,also. Unfortunately it didn't work : it says kernels and modules mismatch.
 
From the ML :

I remember having read in the book The Design and Implementation of the FreeBSD® Operating System Second Edition by Marshall Kirk McKusick, George V. Neville-Neil, and Robert N.M. Watson, that FreeBSD has the ability of using drivers compilers for previous versions of FreeBSD by an stable ABI. (Which also kind of implies forward compatibility of drivers). My question is this true ? And if true. where is the documentation of such ABI? And I can statically link to the kernel one of this portable kernels?
 
ziomario said:
From the ML :

I remember having read in the book The Design and Implementation of the FreeBSD® Operating System Second Edition by Marshall Kirk McKusick, George V. Neville-Neil, and Robert N.M. Watson, that FreeBSD has the ability of using drivers compilers for previous versions of FreeBSD by an stable ABI. (Which also kind of implies forward compatibility of drivers). My question is this true ? And if true. where is the documentation of such ABI? And I can statically link to the kernel one of this portable kernels?
Click to expand...

It's not true. FreeBSD has a stable kernel API, but not ABI. As you can see.

So do I understand that correctly that 14.0 is working correctly for you? Then why did you need for somebody to give you the module?
 
  • Like
Reactions: cy@
cracauer@ said:
So do I understand that correctly that 14.0 is working correctly for you? Then why did you need for somebody to give you the module?
Click to expand...

I have two FreeBSD systems on two different disks. On the first disk (ZFS) there is 14.0 where the GPU passhthru works great. On another disk (UFS) I have upgraded 14.0 to 14.1 (using the freebsd-update script). Here the vmm.ko module that works on 14.0 does not work.
 
ziomario said:
I have two FreeBSD systems on two different disks. On the first disk (ZFS) there is 14.0 where the GPU passhthru works great. On another disk (UFS) I have upgraded 14.0 to 14.1 (using the freebsd-update script). Here the vmm.ko module that works on 14.0 does not work.
Click to expand...

Can you clarify? The 14.0 that works, does it work with stock 14.0 or does it require the module you've been sent?
 
At the beginning he sent to me his personal bhyve files,located on /usr/sbin and called bhyve* and the vmm.ko file and magically I saw that the GPU passthru worked. I thought that those files were specials in some way. But then I found them on the stock 14.0 and when I have reinstalled the stock 14.0 on another disk (using the official DVD image) I have seen that the GPU passthru still worked. When I have upgraded 14.0 to 14.1 gpu passthru stopped working.

Today I have conducted more experiments. I've asked to him to send again his bhyve files (bhyve* files located on /usr/sbin) and his vmm.ko files and his whole /boot directory,everything recompiled for the 14.1. His kernel file + his modules didn't work for me :

Untitled.jpg


at this point,I have used my kernel file and my modules (compiled on my machine when I have upgraded 14.0 to 14.1 with the freebsd-update script) and his bhyve files (located under the directory /usr/sbin and his vmm.ko file) ; all this stuff has been accepted by my 14.1 system,but unfortunately the GPU passthru didn't work.

At this point I made another 2 experiments. I have reinstalled 14.1 from the official DVD image. On the first experiment I have used the kernel file and the bhyve files used by default by the 14.1 just installed. Again the gpu passhtru didn't work. On the experiment 2 I have used the kernel file used by default by the 14.1 but with the bhyve files of the reddit user,but again,the GPU passthru didn't work.

I don't know if can be useful,but on the 14.0,where the GPU passthru works,I see that vmm.ko occupy these memory addresses:

Code: 
[root@marietto /home/marietto]==> kldstat | grep vmm
11    1 0xffffffff81fd7000   37c540 vmm-win.ko

but on the 14.1 I see that it occupies different memory addresses :

Code: 
root@marietto:/home/marietto/Desktop # kldstat | grep vmm
6    1 0xffffffff82177000   37c9e8 vmm.ko

maybe this could be the reason why it does work on the 14.0 but not on the 14.1 ? should it occupy the same memory addresses ?
 
So in summary, 14.0 works for your pci passthrough and 14.1 broke it. There is no indication that the files you got from that users were in any way special or different from the vanilla 14.0 files.

If that is accurate that opens the question how PCI passthrough could have become broken without anybody else noticing and opening a big report.

Can you share your config?
 
cracauer@ said:
If that is accurate that opens the question how PCI passthrough could have become broken without anybody else noticing and opening a big report.
Click to expand...

You hit the nail on the head. The gpu passthru for the Windows VMs and the nvidia gpus works for a limited number of users. The special thing is that it is working for me at least on the 14.0. The developers say that they don't understand fully why in most cases Windows reports error 12 or the error 39. So,the reason why it works is buried between the patches that have been developed for the stock 14.0 and maybe another patch developed for the 14.1 broke something on the 14.0. Or maybe the secret can be found with the reverse engineering of the vmm.ko file ? or by exploring which source code or patches have been used ? I guess that looking for the patch that broke the functionality is like looking for a needle in a haystack.

---> Can you share your config ?

Which config do you need ?
 
ziomario said:
You hit the nail on the head. The gpu passthru for the Windows VMs and the nvidia gpus works for a limited number of users. The special thing is that it is working for me at least on the 14.0. The developers say that they don't understand fully why in most cases Windows reports error 12 or the error 39. So,the reason why it works is buried between the patches that have been developed for the stock 14.0 and maybe another patch developed for the 14.1 broke something on the 14.0. Or maybe the secret can be found with the reverse engineering of the vmm.ko file ? or by exploring which source code or patches have been used ? I guess that looking for the patch that broke the functionality is like looking for a needle in a haystack.

---> Can you share your config ?

Which config do you need ?
Click to expand...

Well, programmers are used to such haystacks. A binary search bisecting src between 14.0 and 14.1 is doable on a fast machine. There seems to be no reason to reverse engineer anything.

Let's start with your bhyve config.
 
Code: 
/boot/loader.conf :

vmm_load="YES"
pptdevs="2/0/0 2/0/1 2/0/2 2/0/3"

/bhyve/12-Win-11-vm12.sh :

/usr/sbin/./bhyve -S -c sockets=4,cores=2,threads=1 -m 8G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/11/Windows11.img,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap12 \
-s 29,fbuf,tcp=0.0.0.0:5912,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:12 < /dev/null & sleep 2 && vncviewer 0:12
 
oh my god :

WARNING: FreeBSD 14.0-RELEASE-p6 is approaching its End-of-Life date. It is strongly recommended that you upgrade to a newer release within the next 2 months.

Should I upgrade to 14.1 within the next 2 months ? I have two months to find the solution.
 
If you are sure that it is working on a regular 14.0-RELEASE, then you could either use git-bisect to find the commit that did the regression or attempt to revert any patch from 14.0 to 14.1 that change anything related to bhyve.
In both case, this could result in some security risk (more security risk for the latter I would assume) and instability.
For the following, I would suggest to do that with an empty state system: new drive (external would be easy), remove the current drive on the machine.
Having a secondary machine for the build is in my opinion necessary, this would allow to build on one machine, generate an image and directly boot on the test machine the new image.
For git-bisect, you could do like that:
Go to the releng/14.1 branch, and take a note of the commit for 14.0.
Then do
Code: 
git bisect start
git bisect bad
git bisect good COMMIT_OF_14.0
Then every time it stop, do a full build, install, and test your setup.
If it is good, do git bisect good
Else, do git bisect bad
 
Hello everyone,

I am the person about whom ziomario sometimes says that I gave him some files, and I'm happy to write my first message on this forum.

To avoid any further confusion, I'd just like to clarify that all the files I sent to ziomario were taken from the original, unmodified FreeBSD 14.0, and later 14.1, base files. Nothing I sent him contained any patches, except for one time, which is not relevant here.
 
Ciao. It's a pleasure for me to see you here. I've said the same as you on post-661184,so everything is good :D. Is the GPU passthru working for you on 14.1 ?
 
ziomario said:
Ciao. It's a pleasure for me to see you here. I've said the same as you on post-661184,so everything is good :D. Is the GPU passthru working for you on 14.1 ?
Click to expand...
:)

GPU passthru working for me on 14.1 for a FreeBSD guest, I didn't have time to check Windows or Linux yet. I believe they will work just fine, as before.
 
You must log in or register to reply here.
Back
Top