Solved Trying to understand why only some kind of vmm.ko module allows to passthru my nvidia GPU to a WIndows 11 vm

ziomario said:
Can you clarify this point ? I don't understand how to go on the releng/14.1 branch and how to take a note of the commit for 14.0,thanks.
Click to expand...
Assuming that you have already cloned the freebsd src repository, you could do the following.
Code: 
git switch main
git fetch origin releng/14.0
git fetch origin releng/14.1
git log -n 1 --pretty=format:%h origin/releng/14.0
git log -n 1 --pretty=format:%h origin/releng/14.1
Which gave me for 14.0 : 5e23806790ef , and for 14.1 : dcdea9e8623e
 
bileslav said:
:)

GPU passthru working for me on 14.1 for a FreeBSD guest, I didn't have time to check Windows or Linux yet. I believe they will work just fine, as before.
Click to expand...

I have some doubt that it works for you on Windows 11 as guest using FreeBSD 14.1. I've reinstalled FreeBSD 14.1 from scratch using the official DVD image and I've checked. It didn't work. I've used the same Windows 11 img file that I used on FreeBSD 14.0. Then I've booted 14.0 again using the same Windows 11 img file and I saw that it works.
 
monwarez said:
Assuming that you have already cloned the freebsd src repository, you could do the following.
Code: 
git switch main
git fetch origin releng/14.0
git fetch origin releng/14.1
git log -n 1 --pretty=format:%h origin/releng/14.0
git log -n 1 --pretty=format:%h origin/releng/14.1
Which gave me for 14.0 : 5e23806790ef , and for 14.1 : dcdea9e8623e
Click to expand...

but the patches should not be delimited and referred to bhyve only ?

Maybe there is an easier method to understand what's the patch that breaks the function. Where FreeBSD saves the patches that it downloads ? I could apply the patches downloaded for the 14.1 to the 14.0 one by one and check if it still works. it could be easier because in this way I can skip playing with git.
 
These patches downloaded by freebsd-update are a differential between 14.0 and 14.1, not an history between 14.0 and 14.1. There is no guarantee that you could infer from it at which point the regression appear.
Using git-bisect allow you to quickly found the culprit commit (the number of commit to test will be as low as the optimal solution to the problem of guessing a number from 0 to N with the hint that the guessed number is lower or greater than the number to found).
 
ziomario said:
Maybe there is an easier method to understand what's the patch that breaks the function. Where FreeBSD saves the patches that it downloads ? I could apply the patches downloaded for the 14.1 to the 14.0 one by one and check if it still works. it could be easier because in this way I can skip playing with git.
Click to expand...

See the problem here is Windows. I would actually go on a crusade to bisect this (if 14.0 worked for me and 14.1 didn't), but I don't want to deal with Windows and its useless error messages. "Didn't work code 42" is not actionable.

I can't recommend doing this with binary patches, in fact it almost certainly won't work.
 
monwarez said:
Assuming that you have already cloned the freebsd src repository, you could do the following.
Code: 
git switch main
git fetch origin releng/14.0
git fetch origin releng/14.1
git log -n 1 --pretty=format:%h origin/releng/14.0
git log -n 1 --pretty=format:%h origin/releng/14.1
Which gave me for 14.0 : 5e23806790ef , and for 14.1 : dcdea9e8623e
Click to expand...

Code: 
[root@marietto /usr]# uname -a

FreeBSD marietto 14.0-RELEASE-p6 FreeBSD 14.0-RELEASE-p6 #0: Tue Mar 26 20:26:20 UTC 2024     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

[root@marietto /usr]# git clone https://git.FreeBSD.org/src.git /usr/src

Clone in '/usr/src' in corso...
remote: Enumerating objects: 4513103, done.
remote: Counting objects: 100% (383516/383516), done.
remote: Compressing objects: 100% (28936/28936), done.
remote: Total 4513103 (delta 377788), reused 354580 (delta 354580), pack-reused 4129587 (from 1)
Ricezione degli oggetti: 100% (4513103/4513103), 1.64 GiB | 10.17 MiB/s, fatto.
Risoluzione dei delta: 100% (3587926/3587926), fatto.
Aggiornamento dei file in corso: 100% (101043/101043), fatto.

[root@marietto /usr]# cd src
[root@marietto /usr/src]# git checkout releng/14.1

Aggiornamento dei file in corso: 100% (22564/22564), fatto.
branch 'releng/14.1' set up to track 'origin/releng/14.1'.
Si è passati a un nuovo branch 'releng/14.1'

[root@marietto /usr/src]# git switch main
Aggiornamento dei file in corso: 100% (22564/22564), fatto.
Si è passati al branch 'main'
Il tuo branch è aggiornato rispetto a 'origin/main'.

[root@marietto /usr/src]# git fetch origin releng/14.0
Da https://git.FreeBSD.org/src
* branch                      releng/14.0 -> FETCH_HEAD

[root@marietto /usr/src]# git fetch origin releng/14.1
Da https://git.FreeBSD.org/src
* branch                      releng/14.1 -> FETCH_HEAD

[root@marietto /usr/src]# git log -n 1 --pretty=format:%h origin/releng/14.0
5e23806790ef

[root@marietto /usr/src]# git log -n 1 --pretty=format:%h origin/releng/14.1
dcdea9e8623e
 
monwarez said:
If you are sure that it is working on a regular 14.0-RELEASE, then you could either use git-bisect to find the commit that did the regression or attempt to revert any patch from 14.0 to 14.1 that change anything related to bhyve.
In both case, this could result in some security risk (more security risk for the latter I would assume) and instability.
For the following, I would suggest to do that with an empty state system: new drive (external would be easy), remove the current drive on the machine.
Having a secondary machine for the build is in my opinion necessary, this would allow to build on one machine, generate an image and directly boot on the test machine the new image.
For git-bisect, you could do like that:
Go to the releng/14.1 branch, and take a note of the commit for 14.0.
Then do
Code: 
git bisect start
git bisect bad
git bisect good COMMIT_OF_14.0
Then every time it stop, do a full build, install, and test your setup.
If it is good, do git bisect good
Else, do git bisect bad
Click to expand...

so,what should I do is :

a) Go to the releng/14.1 branch :

# git fetch origin releng/14.1
Da https://git.freebsd.org/src
* branch releng/14.1 -> FETCH_HEAD
OK

b) take a note of the commit for 14.0 :

[root@marietto /usr/src]# git log -n 1 --pretty=format:%h origin/releng/14.0
5e23806790ef

5e23806790ef is the commit of 14.0. I took note of it.

c) now should I do :

git bisect start

?

---> do a full build, install :

do a full build,install : how ? with which commands ?

# make -j12 buildworld
# make -j12 kernel

are good ?
 
I'm making some mistakes :

Code: 
[root@marietto /usr/src]==> git bisect start 5e23806790ef
You are already on 'main'
Your branch is updated respect to 'origin/main'.
status: waiting for good commit(s), bad commit known

[root@marietto /usr/src]==> git bisect bad dcdea9e8623e
status: waiting for good commit(s), bad commit known

[root@marietto /usr/src]==> git bisect good 5e23806790ef
Bisection: a merge base must be tested

[dde83fc4d2e6e8a3009bdfb7d14dcc8ef802b22f] pci_iov: Refuse to create VFs which require ARI if ARI is not availabl
 
You then need to build it with this version and test if this is working.
A big word of caution: do not do the installation on an existing system, use some external drive and maybe remove the existing one.
From this step you should do a full build and generate an installation image, then use it to test if the virtualization is working.
Ideally you would use one system to do the build, and another for the testing.
After doing the buildworld and buildkernel, you could follow the instruction from https://klarasystems.com/articles/building-customized-freebsd-images/
for creating an image from it.
 
Very thanks. Its doing make -j12 buildworld / make -j12 kernel and then I will follow the instructions on the link you have provided. Please tell me : after building the image,should I repeat all the steps from the beginning ? and maybe I should try until I find that the gpu is not able to be passed ? how can I know what is the wrong patch ?
 
I have to disagree with the advise.

I would make a solid backup of your machine now and then modify it directly with `make installworld installkernel`. Making an image and installing it slows you down too much.

You are only messing around within the confinements of 14-stable, so it isn't too likely that you crash and burn.
 
cracauer@ said:
I have to disagree with the advise.

I would make a solid backup of your machine now and then modify it directly with `make installworld installkernel`. Making an image and installing it slows you down too much.

You are only messing around within the confinements of 14-stable, so it isn't too likely that you crash and burn.
Click to expand...

I need to know how much times I should repeat the procedure,to understand how much time I need more or less to find the wrong patch and remove it ; anyway I have a powerful machine.
 
ziomario said:
I need to know how much times I should repeat the procedure,to understand how much time I need more or less to find the wrong patch and remove it.
Click to expand...

That depends on how lucky you are :)

How many commit do you want to try? Did you make a first pass identifying commits that touch the components in question?
 
cracauer@ said:
That depends on how lucky you are :)

How many commit do you want to try ?
Click to expand...

I don't have clear ideas. How many commit I want to try ? I didn't understand how it works. I think that I want to try until I find a version that does not work. What I haven't understood is how can I identify the patch that broke the gpu passthru.

Did you make a first pass identifying commits that touch the components in question ?
Click to expand...

I don't know. Following the moonwarez instructions , I wrote what I did by opening a bug ticket :

280161 – virtualization/bhyve: gpu passthru on Windows works on the 14.0,but stopped on 14.1 : looking for a regression...

bugs.freebsd.org bugs.freebsd.org

So,I'm not able to reply to your question. So,please elaborate more. I need to understand how is the workflow.
 
Well, you are the one testing each FreeBSD version during the search and determine whether passthrough works the way you want. I assume you start your Windows VM for that and see what happens.
 
cracauer@ said:
Well, you are the one testing each FreeBSD version during the search and determine whether passthrough works the way you want. I assume you start your Windows VM for that and see what happens.
Click to expand...

I have understood this. What I didn't understand is what happens next. Lets assume that I found a version that does not work,how can I know to which commit it belongs ? In other words I should identify,in some way,the wrong patch,or not ? Because I think that then I should inform the developers that that commit introduced a regression that stopped the functionality.
 
ziomario said:
I have understood this. What I didn't understand is what happens next. Lets assume that I found a version that does not work,how can I know to which commit it belongs ? In other words I should identify,in some way,the wrong patch,or not ?
Click to expand...

You go by commits anyway, e.g. 2abea9df01655633aabbb9bf3204c90722001202

You get the diff for that commit with

Code: 
git diff 2abea9df01655633aabbb9bf3204c90722001202^!

You reverse apply that diff to 14.1 with `patch -R`. That should give you a 14.1 that works for you.
 
cracauer@ said:
You go by commits anyway, e.g. 2abea9df01655633aabbb9bf3204c90722001202

You get the diff for that commit with

Code: 
git diff 2abea9df01655633aabbb9bf3204c90722001202^!

You reverse apply that diff to 14.1 with `patch -R`. That should give you a 14.1 that works for you.
Click to expand...

man,I suffer a lot because you explain things in a strangled way....

How can I obtain the number 2abea9df01655633aabbb9bf3204c90722001202 ?
 
ziomario said:
How can I obtain the number 2abea9df01655633aabbb9bf3204c90722001202 ?
Click to expand...

You use `git log origin/stable/14`. You select those commits between the release dates of 14.0 and 14.1. Collect the hashes in a file.

You then use `git checkout <hash>` to test one of them. You should test the first and the last one first to make sure one is working and one is broken. Then you do a binary search on the commit hashes in between. Do you know what a binary search is?
 
ziomario said:
I have some doubt that it works for you on Windows 11 as guest using FreeBSD 14.1. I've reinstalled FreeBSD 14.1 from scratch using the official DVD image and I've checked. It didn't work. I've used the same Windows 11 img file that I used on FreeBSD 14.0. Then I've booted 14.0 again using the same Windows 11 img file and I saw that it works.
Click to expand...
Hi. I've just checked Windows 10, please see the screenshot. Windows 11 never worked for me, which I mentioned several times:
Due to a bug, it's currently impossible to allocate more than 1 vCPU to Windows 11 when doing passthrough. Details here.
Click to expand...

Screenshot_2024-07-07_08-56-31.png
 
Here's the config of that VM:
sh: 
loader="uefi"
uefi_vars="yes"
uuid="e251f8bd-3c27-11ef-8eaa-a8a159344963"
bhyve_options="-A"

ahci_device_limit="8"
utctime="no"

cpu="12"
cpu_sockets="1"
cpu_cores="12"
cpu_threads="1"
memory="16G"

network0_type="virtio-net"
network0_switch="public"
network0_mac="58:9c:fc:05:03:c5"

disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"

passthru0="10/0/0=10:0"
passthru1="10/0/1=10:1"
 
P. S. I have not applied Corvin's patches for Linux guest NVIDIA support in this 14.1 installation.

I'm actually not sure, maybe they are upstreamed it in 14.1-RELEASE? Because I don't see a branch like origin/phab/corvink/14.1/nvidia-wip in their repo. Only that old origin/phab/corvink/14.0/nvidia-wip. Maybe "14.0" here means the whole branch (14-STABLE?), so it's compatible to compile in 14.1-RELEASE; I don't know.

If the patches included in 14.1, it may be the reason you face different behavior, since you said that they are incompatible with Windows for you (which is not the case for me).

I need to try boot a Linux VM to prove this theory, but I don't have time for that yet. Until then, probably someone here can tell if those patches were upstreamed or not.
 
You must log in or register to reply here.
Back
Top