Can't kldload nvidia-drm

G

GlenH

I'm trying to fix an ongoing issue with not being able to run Wayland. In reading up on the issue it appears that you need to compile the nvidia-drm driver from the port because the version in the package repository doesn't match up to the current kernel. So I did a git pull and built it from source. I had to manually move the existing /boot/modules/nvidia-drm.ko in order to get make to recognize that it needed to actually build.

I now get the following error:

# sudo kldload nvidia-drm.ko
kldload: can't load nvidia-drm.ko: module already loaded or in kernel
# kldstat -v | grep nvidia
78 pci/ata_nvidia
360 g_raid_md_nvidia
4 2 0xffffffff83200000 535f868 nvidia.ko (/boot/modules/nvidia.ko)
508 vgapci/nvidia
7 1 0xffffffff8305e000 1517d8 nvidia-modeset.ko (/boot/modules/nvidia-modeset.ko)
509 nvidia-modeset
Click to expand...

so it clearly isn't in the kernel, yet kldload tells me that it is. Can anyone tell me what's going on here?

FWIW:

# freebsd-version -kru
14.3-RELEASE
14.3-RELEASE
14.3-RELEASE-p1
% uname -a
FreeBSD glen-server 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64
Click to expand...
 
OK, I think I solved my own problem. For anyone's future reference: this problem occurs when you've built a kernel module from source but the kernel source in /usr/src isn't the same as the kernel that's actually running. So kldload is lying: it's saying the problem is that the kernel module is already loaded, when the real problem is that the kernel module is compiled against the wrong version of the kernel.

To switch to the right version of the kernel, do:

# cd /usr/src/
# git checkout releng/14.3
Click to expand...

or whatever version of the kernel is reported by uname -r

Then manually remove the .ko from /boot/modules/, then rebuilt and reinstall the module from the appropriate port.
 
Usually, x11/nvidia-driver[-304|-340|-390|-470-devel] are "relatively" robust with mis-match between kernel and kmods.
This is because these depends on FreeBSD KPI/KBI, which is "relatively" stable within single major version.

On the other hand, graphics/nvidia-drm-[510|515|61|66]-kmod[-devel] are quite, quite and quite fragile, because these depends on toooooo fragile (as of upstream) LinunKPI, which in many cases every single commits to them causes breakages, thus, needs rebuilds. So as graphids/drm-[510|515|61|66]-kmod.

So I strongly recommend to manage your /usr/src tree with devel/git, to checkout exactly same commit which the running kernel is built against, and rebuild/reinstall all ports kmods which requires LinuxKPI every time you upgrade kernel. For reference, PR 288314 and PR 288565.
 
what's the output of,
freebsd-version -kru
It's should give three times the same.
And for /usr/ports with git checkout the correct branch.
 
For confirming version for LinuxKPI consumers, checking dmesg(8) is better.
Around 5 lines under a line marked ---<<BOOT>>---, 12 digits commit hash should be shown even on WITH_REPRODUCIBLE_BUILD, that is the default at least for *-Release.
 
Adventures continue.

When the day started, the nvidia-drm module would load, but none of the Wayland compositors I tried would come completely up. Plasma6 would give me a black screen, and after about thirty seconds would attempt to dump me back to the console, except that the console VGA driver was hosed -- I would get colored gibberish text blocks on the screen, and would need to reboot the system to get it back.

Wayfire would bring me to a grey screen with a mouse pointer, but no menu bar or anything else.

X11 worked fine, and I took that as a decently good sign. Except that when I terminated the X11 window manager and went back to the console it was similarly hosed to the above.

After five or six reboots, the system started to hang when the boot loader loaded the nvidIa-drm module. The screen would go black and HDMI output would terminate. Eventually I had to boot into single user mode and remove nvidia-drm from the kld_list.

Any suggestions at this point would be appreciated.
 
Alain De Vos said:
what's the output of,
freebsd-version -kru
It's should give three times the same.
And for /usr/ports with git checkout the correct branch.
Click to expand...

As follows:

# freebsd-version -kru
14.3-RELEASE
14.3-RELEASE
14.3-RELEASE-p1
% uname -a
FreeBSD glen-server 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64

# git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
Click to expand...
 
elgrande said:
As GlenH said you need to git checkout releng/14.3.
Click to expand...
I'm not entirely sure how to do that, as the releng/14.3 branch doesn't appear to exist:

# git branch -a
* main
remotes/origin/2014Q1
remotes/origin/2014Q2
remotes/origin/2014Q3
remotes/origin/2014Q4
remotes/origin/2015Q1
remotes/origin/2015Q2
remotes/origin/2015Q3
remotes/origin/2015Q4
remotes/origin/2016Q1
remotes/origin/2016Q2
remotes/origin/2016Q3
remotes/origin/2016Q4
remotes/origin/2017Q1
remotes/origin/2017Q2
remotes/origin/2017Q3
remotes/origin/2017Q4
remotes/origin/2018Q1
remotes/origin/2018Q2
remotes/origin/2018Q3
remotes/origin/2018Q4
remotes/origin/2019Q1
remotes/origin/2019Q2
remotes/origin/2019Q3
remotes/origin/2019Q4
remotes/origin/2020Q1
remotes/origin/2020Q2
remotes/origin/2020Q3
remotes/origin/2020Q4
remotes/origin/2021Q1
remotes/origin/2021Q2
remotes/origin/2021Q3
remotes/origin/2021Q4
remotes/origin/2022Q1
remotes/origin/2022Q2
remotes/origin/2022Q3
remotes/origin/2022Q4
remotes/origin/2023Q1
remotes/origin/2023Q2
remotes/origin/2023Q3
remotes/origin/2023Q4
remotes/origin/2024Q1
remotes/origin/2024Q2
remotes/origin/2024Q3
remotes/origin/2024Q4
remotes/origin/2025Q1
remotes/origin/2025Q2
remotes/origin/2025Q3
remotes/origin/HEAD -> origin/main
remotes/origin/main
# sudo git switch releng/14.3
fatal: invalid reference: releng/14.3
Click to expand...
 
So I switched to the git tag "release/14.3.0", as that seemed like the most likely match to releng/14.3. But the resulting nvidia-drm.ko won't load:

# sudo kldload nvidia-drm
kldload: an error occurred while loading module nvidia-drm. Please check dmesg(8) for more details.
# dmesg
link_elf_obj: symbol __drm_dbg undefined
linker_load_file: /boot/modules/nvidia-drm.ko - unsupported file type
Click to expand...

Also, the version of nvidia-drm that it actually makes is 510, not (as I would otherwise expect) 61.
 
Clearly, you're mangling /usr/src and /usr/ports.

As of your outputs in Comment #9, your kernel is GENERIC on branch releng/14.3 at commit 8c9ce319fef7, unmodified.

And your git operations in Comment #11 shows you're operating on /usr/ports, while elgrande meant for /usr/src.

If you've installed latest pkgs instead of default quarterly (now 2025Q3) or building everything with ports locally using main branch, switch /usr/ports back to main branch first. If you were using quarterly, to 2025Q3, not main.

Then, on /usr/src, checkout commit 8c9ce319fef7. If not yet cloned, clone repo first. Then build/install all ports which installs any kmod (*.ko).
 
T-Aoki said:
For confirming version for LinuxKPI consumers, checking dmesg(8) is better.
Around 5 lines under a line marked ---<<BOOT>>---, 12 digits commit hash should be shown even on WITH_REPRODUCIBLE_BUILD, that is the default at least for *-Release.
Click to expand...

This is what I'm running:

FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64
Click to expand...

...and at this point my ignorance of git is starting to hurt me. How do I check out a specific commit hash?
 
git checkout 8c9ce319fef7
In general i recommend not using checkout of specific commits on regular boxes as with time you may run into difficulties such as you have. they are for testing things and so on. After checkout rebuild the ports that are indtalling kernel modules as pointed above.
 
T-Aoki said:
Clearly, you're mangling /usr/src and /usr/ports.

As of your outputs in Comment #9, your kernel is GENERIC on branch releng/14.3 at commit 8c9ce319fef7, unmodified.

And your git operations in Comment #11 shows you're operating on /usr/ports, while elgrande meant for /usr/src.

If you've installed latest pkgs instead of default quarterly (now 2025Q3) or building everything with ports locally using main branch, switch /usr/ports back to main branch first. If you were using quarterly, to 2025Q3, not main.

Then, on /usr/src, checkout commit 8c9ce319fef7. If not yet cloned, clone repo first. Then build/install all ports which installs any kmod (*.ko).
Click to expand...

OK, so, that gets me back to the previously reported behavior.

I switched ports back to main. I switched src to commit hash 8c9ce319fef7 by issuing git checkout 8c9ce319fef7, and now I get:

# git status
HEAD detached at 8c9ce319fef7
nothing to commit, working tree clean
Click to expand...

The nvidia-drm module builds and loads. X11 works. Plasma6 and Wayfire don't. Trying to start any display server, X11 or Wayland, screws up the VGA driver, and I have to reboot to get a usable text console back.
 
angry_vincent said:
git checkout 8c9ce319fef7
In general i recommend not using checkout of specific commits on regular boxes as with time you may run into difficulties such as you have. they are for testing things and so on. After checkout rebuild the ports that are indtalling kernel modules as pointed above.
Click to expand...

Yeah. I only went down this path because there is no nvidia-drm package for 14.3
 
T-Aoki said:
Are you sure you restarted your computer after reinstalling kmods?
Click to expand...

Yes. I just did it again after a reboot, just to make sure.

T-Aoki said:
If I recall correctly, at least some of kmods installed by graphics/drm-*-kmod hesitates to be kldunload'ed, thus, restart is needed for rebuilt kmods to be loaded.

And are you sure you have hw.nvidiadrm.modeset=1 line in your /boot/loader.conf as noted on messages shown when installation of graphics/nvidia-drm-*-kmod?
Click to expand...

Yes.

T-Aoki said:
(I assume you've installed graphics/nvidia-drm-61-kmod via metaport graphics/nvidia-drm-kmod.)
Click to expand...

Yes. To be entirely clear, I cd to /usr/ports/graphics/nvidia-drm-kmod and issue a "sudo make clean" followed by "sudo make install". This downloads some blobs, does a bunch of compiling, and the end result is a new nvidia-drm.ko in /boot/modules/.

Right now I am not adding nvidia-drm to kld_list in /etc/rc.conf, because if it fails to load it puts me into a kernel panic that I have to unwind via the boot loader prompt, which is a pain. I am issuing a "sudo kldload nvidia-drm" from the console, and then (for instance) starting Plasma via "exec dbus-start --exit-with-session ck-launch-session startplasma-wayland"
 
Is your nvidia GPU any of RTX50*?
If so, try adding hw.nvidia.registry.EnableGpuFirmware=1 line in your /boot/loader.conf. RTX50* series are known not working properly without GSP firmware loaded. Related commit here. And there's some issues for RTX50* which cannot be fixed by FreeBSD side (need to be fixed by nvidia side). See PR 287453 if you're insterested.

But I assume no, as you can start X11.
 
T-Aoki said:
Is your nvidia GPU any of RTX50*?
If so, try adding hw.nvidia.registry.EnableGpuFirmware=1 line in your /boot/loader.conf. RTX50* series are known not working properly without GSP firmware loaded. Related commit here. And there's some issues for RTX50* which cannot be fixed by FreeBSD side (need to be fixed by nvidia side). See PR 287453 if you're insterested.

But I assume no, as you can start X11.
Click to expand...

It's a GeForce GTX 1050 TI

Note that X11 works just fine with nvidIa-modeset.ko, and this does not mess with the text console. It's just nvidia-drm that screws things up.
 
Are you sure graphics/drm-61-kmod is rebuilt/reinstalled, too?
  • graphics/nvidia-drm-kmod pulls in graphics/nvidia-drm-61-kmod for 14.3-Release by default,
  • graphics/nvidia-drm-61-kmod depends on both graphics/drm-61-kmod and x11/nvidia-driver.
So if graphics/drm-61-kmod is not updated (as it could be considered to be up-to-date), it causes mis-match.
 
You must log in or register to reply here.
Back
Top