Nvidia Optimus Driver for FreeBSD

Hi, I made a modified version of x11/nvidia-driver that works with optimus laptops/devices (muxless). It provides an optimus service (managing extra X server we need, nvidia modules, configs and etc) and a optirun command to run programs on the nvidia gpu:

6725


Right now I could only test it on my own laptop (ASUS G56JK), it's a muxless haswell + gtx850m, running FreeBSD 12, nvidia drivers version 390 and 430 both work perfectly. If anyone is interested in testing and/or helping feel free to leave a message or contact me. You can find the Makefile and binary releases here: https://github.com/pouya-eghbali/freebsd-nvidia-optimus

If anyone has any questions or needs help to make this work on their devices, I'll be glad to help. If you try this and it works for you, sharing your hardware info is appreciated and can probably help others. Please note this is a pre-release.
 
If anyone has any questions or needs help to make this work on their devices, I'll be glad to help.
Have a look through https://bugs.freebsd.org/bugzilla/ for open PRs regarding this. I'm fairly certain there are one or more open cases about this. If you can't find an open case, open one yourself. Help out there and we all benefit. You do need to register but anyone can respond, provide patches or help out in any other way.
 
Have a look through https://bugs.freebsd.org/bugzilla/ for open PRs regarding this. I'm fairly certain there are one or more open cases about this.

I'm aware of an open PR that has been a work in progress since 2014, that's a long time and I guess the patches provided there won't work anymore and one of the solutions there makes the intel gpu unusable when we run something on the nvidia gpu. All this, I decided to start my own version. Another issue is the version of the nvidia driver, there are open PRs (which I used to patch my Makefile and make it v430 ready) but it's been a long time since they're posted and they're not merged yet.

I will definitely respond to these PRs and provide my patches so at least people can study them.
 
I will definitely respond to these PRs and provide my patches so at least people can study them.
Yes, please do. And yes, leave the really old ones, they're likely too old to be useful. The bug tracking system we have isn't perfect and sometimes things get a little cluttered.
 

No I didn't know of it, I reviewed what they have done and it's very similar to what I did (this is basically how bumblebee works on Linux). I searched a lot to see if there's an easy solution to this but couldn't find, surprisingly my empty 'freebsd nvidia' repository is result #4 when if you google 'freebsd nvidia optimus'. Well, after 2 years I returned to freebsd and decided to make this work for real.
 
surprisingly my empty 'freebsd nvidia' repository is result #4 when if you google 'freebsd nvidia optimus'. Well, after 2 years I returned to freebsd and decided to make this work for real.

Interesting… I didn't realize there was another repository.

Now, regarding your (current) work:

1. Bundling everything together (in one single commit, even) makes it difficult to see the changes. If you really must, then keep the port files unchanged for the initial commit and commit your work on top of that.

2. "Nvidia Optimus Driver" makes it sound like it is the definitive solution to that problem, while in reality besides VirtualGL there are also Primus and Primus-VK as well as reverse PRIME GPU offloading, which is the closest thing to the "real" Optimus. Although I am unsure to what extent PRIME is actually usable.
 
1. Bundling everything together (in one single commit, even) makes it difficult to see the changes. If you really must, then keep the port files unchanged for the initial commit and commit your work on top of that.

That's why I still haven't made a PR to the nvidia driver port. I didn't want to do all this for v390 driver, I wanted the latest 430 so I had to patch the nvidia driver makefile, otherwise it's possible to provide an optimus option and do a slave port, or just do a slave port and move the files after build is finished.

2. "Nvidia Optimus Driver" makes it sound like it is the definitive solution to that problem, while in reality besides VirtualGL there are also Primus and Primus-VK as well as reverse PRIME GPU offloading, which is the closest thing to the "real" Optimus. Although I am unsure to what extent PRIME is actually usable.

I'm aware of these, bumblebee on linux has several backends, virtualgl was the default but they added primus because virtualgl has some performance issues. However, primus was last updated 7 years ago and it's not available for freebsd, maybe we can make it work on freebsd but I'm not really willing to port a dead project. About PRIME, it uses randr, might be possible under freebsd, I'll do some experiments on that.

What I prefer, is adding optimus option to nvidia driver port, making a slave port for nvidia optimus (which will install the required nvidia libraries), and have some optimus-utilities-${backend} ports where backend can be virtualgl, primus or prime or whatever else that works.
 
Usually I communicate my freebsd-related work on the mailing lists and bug trackers, but I've just registered to the forum as well.

I'd like to bring to your attention a similar effort I began over a year ago.

Now the situation that we have both worked on this and not shared efforts so far is entirely my fault; however I hope we can combine the efforts for the best way forward.

Version 0.1 of freebsd-gpu-headless was two parts:
- A slave port of nvidia-driver, called nvidia-headless-driver, with no changes other than file paths fixed up to avoid breaking Mesa GLX.
- A set of scripts, called nvidia-headless-utils, with nvidia-xconfig wrapper to ease configuration, and virtualgl wrapper to enable rendering to integrated GPU display.
Support was only basic, and required a bit of manual configuration. I never got this version polished well enough to submit it into FreeBSD, but maybe it would have been good enough. I did a very poor job of publicizing the work, its only mention is at https://lists.freebsd.org/pipermail/freebsd-x11/2018-March/020675.html

Since then (for the last year) I have had a "version 0.2" as local changes to my installation, with working nvidia-xconfig, ondemand power switching with acpi_call, and Linux support (both 32- and 64-bit). Unfortunately I did not have sufficient time and motivation to give this the attention it deserved towards making it official.

Realizing, as I did this last weekend, that others are moving ahead on this and potentially duplicating this effort, I've taken the time to wrap this up and publish a version 0.3. It is essentially ready to be submitted for inclusion in Ports, pending your feedback.


Some parts I have had working are not quite ready, including the Linux support and power switching. The rest is there, including the rc.d script for running Nvidia in the background. In fact keeping Xorg attached achieves the majority of the power savings; as it prevents the GPU from spinning and wasting power. The consumption difference in my Dell XPS between running with Xorg idle vs. disabling the device by ACPI is only ~0.5 Watt whereas leaving it in its power-on state, kernel module loaded or not, with no Xorg, wastes about 5 Watts!

Other things I have addressed:
- User expects nvidia-xconfig to just work. This is accomplished with a very minor change to that port.
- Display number used for Nvidia is configurable, and may be overridden in environment.
- Two ports for two use-cases:
- nvidia-headless-utils: For using Nvidia as a compute resource on a headless server.
- nvidia-hybrid-graphics: For notebook computers with iGPU+dGPU ("Optimus")
- nvidia-headless-driver is just a minimal slave port. The changes to nvidia-driver needed are also minimal and noninvasive. This is very important for the future maintainability.

Since you have gotten this far with your "optimus" port, I gather that you have a good understanding of what is needed, and might have the time to help integrate what I have here. I strongly recommend adopting the approach of breaking it down into small understandable components with the discrete purposes, rather than as a single port for all. I agree with shkhln that having the changes to a single Makefile is hard to follow (and it is all in there alongside the nvidia-driver internals).

I've avoided calling mine "Optimus" (although I do include that name in the description so it can be searched) since, as others have noted, Optimus is the proprietary switching software, something that will never be on FreeBSD unless provided by Nvidia themselves. "Hybrid" seems an acceptable genericized name.

So far, nvidia-hybrid-graphics depends on VirtualGL because currently that is the only working solution.

Real PRIME support would be great!
 
Hi Theron. I wasn't aware of your port, I searched a lot in past (a few years ago) and couldn't find a solution so I decided to make my own, I managed to make it work a few years ago but it wasn't clean. Now I've returned to FreeBSD and I want to stay at all costs (well it's difficult to be a FreeBSD desktop user... but things are getting better).

My current port, is just a quick port to test how things will work, and also I wanted the latest drivers so I had to patch the nvidia driver port, but what I planned was making several different ports for this, possibly with different backends and for different uses (with defaults that make sense and doesn't require the user to do extensive research), I am aware my port isn't perfect, neither are the scripts I wrote.

The idea of having `nvidia-headless-utils`, `nvidia-hybrid-graphics` and `nvidia-headless-driver` is really nice, I like it and it's close to what I had in mind. I'll be glad to work together on this and maintain it together.

About the power savings, I was planning to provide methods to turn on/off the gpu (possibly automatically, before and after we start the X server) but I noticed very strange behavior. For me calling the correct turn off method using acpi_call (I extracted my DSDT to find out the correct method, but there's the famous https://people.freebsd.org/~xmj/turn_off_gpu.sh which does the exact same thing anyways) makes the GPU fan go crazy, I cannot even make it stop by rebooting I have to power off (that's why I decided to not include this in my port).

Anyways, we can share our experience and make a working, perfect solution.
 
Hi Theron. I wasn't aware of your port, I searched a lot in past (a few years ago) and couldn't find a solution so I decided to make my own, I managed to make it work a few years ago but it wasn't clean. Now I've returned to FreeBSD and I want to stay at all costs (well it's difficult to be a FreeBSD desktop user... but things are getting better).

My current port, is just a quick port to test how things will work, and also I wanted the latest drivers so I had to patch the nvidia driver port, but what I planned was making several different ports for this, possibly with different backends and for different uses (with defaults that make sense and doesn't require the user to do extensive research), I am aware my port isn't perfect, neither are the scripts I wrote.

The idea of having `nvidia-headless-utils`, `nvidia-hybrid-graphics` and `nvidia-headless-driver` is really nice, I like it and it's close to what I had in mind. I'll be glad to work together on this and maintain it together.
I thought this was the most sensible thing, and I am glad to hear that you agree. In that case, I think it makes sense to go with those ports and add in your features. Having the daemon load the kernel module is good, since it is one fewer rc.conf entry to require of the user. Modeset vs. non-modeset option is also good, and Xorg server flags might be useful to someone. I have Xorg display number handled by ${LOCALBASE}/etc/nvidia-headless.conf but maybe there is also a legitimate reason to override it from rc.conf; I don't know.

I avoided using "Optimus", "opti", etc. for reasons discussed but there may be significant value in keeping "optirun" name for sake of familiarity to ex- Linux users.

I'd like to find a better name for the daemon than either "nvidia_headless_xorg" (annoying to type) or "optimus".
I do like keeping things general: when a FreeBSD user/admin on a headless Nvidia-powered compute server might find headless-utils useful and not want to worry about anything laptop-related. However, I would imagine that existing nvidia-driver is perfectly adequate in this situation, so maybe this is a silly concern.

About the power savings, I was planning to provide methods to turn on/off the gpu (possibly automatically, before and after we start the X server) but I noticed very strange behavior. For me calling the correct turn off method using acpi_call (I extracted my DSDT to find out the correct method, but there's the famous https://people.freebsd.org/~xmj/turn_off_gpu.sh which does the exact same thing anyways) makes the GPU fan go crazy, I cannot even make it stop by rebooting I have to power off (that's why I decided to not include this in my port).
That is unfortunate. I think I have heard of this issue, in a discussion of MacOS on unlicensed hardware (usually good source of information on DSDT). If I recall correctly, that user's fix was to get the Nvidia driver working rather than power off the GPU. Do you know how Linux and/or Microsoft handles this? Do the other _OFF or similarly named ACPI methods do anything useful?

Anyways, we can share our experience and make a working, perfect solution.
Other than responding to messages, (and wasting some time fiddling with PRIME, see below), I don't know when I will next be able to work on this. If you have time, could you try the freebsd-gpu-headless ports and look into adding the missing features? I probably have time at least to review contributions. Feedback from others would be useful too.

Waiting for PR 232645 makes sense. Does the oldest GPU supported by 430 predate hybrid mobile graphics, or will someone appreciate a 390 or earlier flavor of nvidia-headless?

I believe it is limited to the Linux driver at the moment.
I tried creating an xorg.conf mostly following the documentation relevant to PRIME. Xorg attaches to both GPUs (each in its own Screen, i.e. :0.0 and :0.1) but there aren't any special RandR resources available for outputting from one to the other. Or is that not really how PRIME is supposed to work?
PRIME's performance edge comes from handling the DMAs directly in kernel, yes? Maybe something can be done without explicit cooperation from Nvidia, close to the binary interface between the closed-source Nvidia modules and the rest of the stack. Primus ("PRIME in User Space"?) performed reasonably well for me on Linux, but wouldn't build on FreeBSD toolchain.
 
Does the oldest GPU supported by 430 predate hybrid mobile graphics, or will someone appreciate a 390 or earlier flavor of nvidia-headless?

I have to agree here. With muxless laptops being in vogue since roughly 2012, there likely to be very few models based on Fermi GPUs. Not to mention that with the usual laptop build quality by this point in time I would expect most of them to be mechanically broken into multiple separate laptop pieces.

I tried creating an xorg.conf mostly following the documentation relevant to PRIME. Xorg attaches to both GPUs (each in its own Screen, i.e. :0.0 and :0.1) but there aren't any special RandR resources available for outputting from one to the other.

The Linux documentation mentions DRM, which I believe is provided by nvidia-drm kernel module available on Linux. That's presumably a hard requirement. The module is not that large (~4000 LOC) and it is available with full source code, and it seems to be well suited for porting with linuxkpi. The big question is, of course, who would do that work?

So far there have been only one, quickly abandoned, attempt at porting nvidia-drm. I wouldn't mind trying my hand at it either, despite the lack of kernel development knowledge, very modest C skill and being a lousy programmer in general, but I can't stand crashing my desktop environment every five minutes and getting GPU passthrough working under bhyve (for development) is an even bigger puzzle.

Or is that not really how PRIME is supposed to work?

There are some news going around which make me think I don't really understand what is supported already and what is yet to be supported. See also PRIME and PRIME Synchronization thread on Nvidia's forum.
 
First of all, I want to say thanks to everyone who is looking at this. It's been one of those "if I have time" things that I really wanted to spend time looking at. I have a Thinkpad T420 and a T560 that both have Nvidia graphics and would like to see this work in FreeBSD.

About the power savings, I was planning to provide methods to turn on/off the gpu (possibly automatically, before and after we start the X server) but I noticed very strange behavior. For me calling the correct turn off method using acpi_call (I extracted my DSDT to find out the correct method, but there's the famous https://people.freebsd.org/~xmj/turn_off_gpu.sh which does the exact same thing anyways) makes the GPU fan go crazy, I cannot even make it stop by rebooting I have to power off (that's why I decided to not include this in my port).

Anyways, we can share our experience and make a working, perfect solution.

For me, when I run `turn_off_gpu` script on my T420, it finds an ACPI command that turns off one of my USB buses and appears to leave the NVidia card alone—at least that's what I can see from the dmesg "detach" output. So, it may that these addresses need updating too, but I have not investigated more.

Anyway, I'll keep watching and hope that I might get some time this weekend to try the work out on the T420.
 
For me, when I run `turn_off_gpu` script on my T420, it finds an ACPI command that turns off one of my USB buses and appears to leave the NVidia card alone—at least that's what I can see from the dmesg "detach" output. So, it may that these addresses need updating too, but I have not investigated more.
One way to see all the ACPI methods: acpidump -o dsdt && acpiexec -b "Methods;" dsdt
acpiexec is sysutils/acpica-tools.
Then use https://github.com/Bumblebee-Project/Bumblebee/wiki/ACPI-for-Developers#naming as a guide for names to look for.
The killing of USB by that turn_off_gpu script is an old problem, but I don't remember where I heard of it.
 
Hi again,

Thank you for the post ot the the ACPI methods. I looked and there were some extra items for a Thinkpad W520, which seems close to what I have. I will need to experiment a bit more with that.
In other news, I tried the nvidia_gpu_headless port. It built and installed OK. I had to do some hand adjustments to the config using pouya's xorg.conf.nv (needed to add the ConnectedMonitor option).

However, I can't seem to get anything to show up on the screen. In my regular X session (i.e., the one I start through SDDM), when I run `nvrun glxgears`, I get output on stderr about it syncronizing the rate and a message about the frame rate every 5 seconds, but I get no window. Is there anything special that I'm missing? Does the virtualgl need some sort of configuration?

I can send config and log files if desired.
 
Hi, can you try nvidia-hybrid-graphics port and nvrun-vgl in place of nvrun? That should work with VirtualGL without extra configuration, unless I have missed something that is only on my system.

No window with nvrun is expected: it is using Nvidia as a headless device, so there is nothing to show the window. I should document this better. I use it for GPGPU application where VirtualGL would only contribute a slowdown.

Thanks for helping to test. What does ConnectedMonitor option do? I'd like to bring Pouya's contributions into the freebsd-gpu ports I have.
 
OK. It seems that the nvidia-headless-utils didn't install nvrun-vgl (but it was in the GitHub repo). There were also a couple of other files (nvidia-hybrid*) that also weren't installed. I updated the pkg-plist, rebuild and updated the port and running glxgears with `nvrun-vgl` works. Hooray! Sorry, didn't have a screen shot tool handy though.

I sent you a pull request for the missing files in the pkg-plist.

As for the ConnectedMonitor option, when running nvidia-headless-xconfig it fails when trying to run the server and the error in the log was something like "Can't find a display to connect to." So, I combined the config that was left in /var/cache with Pouya's Xorg config—Basically the ConnectedMonitor and UseEDID options and put it in the /usr/local/etc/X11/xorg-nvidia-headless.conf. Then, I was able to start the server. It could be that later Nvidia cards don't care about this, but at least the one on my machine wanted a monitor to connect to.

It was a little bit fiddly, but it seems to work now. I'll have to examine the acpi stuff at another time. Maybe it will be as you say, that as soon as X11 connects to it, the power drops, we'll see. Regardless, at least I can use the Nvidia card on the T420 without switching things in the BIOS, something that had bothered me for a while, and something to further experiment with. Thank you for the effort. I very much appreciate it and hope that you can get the port committed.
 
Does the oldest GPU supported by 430 predate hybrid mobile graphics, or will someone appreciate a 390 or earlier flavor of nvidia-headless?
I have to agree here. With muxless laptops being in vogue since roughly 2012, there likely to be very few models based on Fermi GPUs. Not to mention that with the usual laptop build quality by this point in time I would expect most of them to be mechanically broken into multiple separate laptop pieces.
My laptop from 2012 still works and is held together by metal pieces I have glued in in the right places...
Evidently the oldest Optimus-capable GPU is a GT 540M: https://www.geforce.com/hardware/technology/optimus/supported-gpus?field_gpu_type_value=All&page=2
It is still supported by 390.87: https://www.geforce.com/drivers/results/137279
Similarly, the 650M in my 2012 laptop (which I do use on occasion) is supported by 390.87 and 418.x but not 430.x.
Therefore someone might appreciate nvidia-headless-driver-390 once nvidia-driver goes to 430 (what is holding up that PR?).
 
Replying here primarily to reassure others that this hasn't been forgotten.

danfe@ says he needs several more days (as of Aug. 3) to move ahead with Nvidia 430.x: PR 232645
In the mean time, I will rebase my nvidia-hybrid Github on Tomoaki AOKI's 430.40 / 390.129 patch.
390.129 uses GLVND but also supports oldest Optimus-capable card, so the nvidia-headless-driver-390 slave port is trivial once I work around the complication of GLVND in 430.40.
That said, I can't be exactly sure when I can next work on this. Forks, patches, and/or pull requests are welcome; I am more likely to have time to at least review these.

GLVND may be useful as an alternative to the way I currently propose to handle conflicting libGL, but really should be done for all GL on FreeBSD, so for now I will keep it discrete so we don't get dragged down with concerns about GLVND from Intel and AMD users who are happy with Mesa as-is.

Next step from there would be to have the Nvidia driver port take responsibility for installing GLVND as the libGL for all apps and without breaking Mesa, but similarly I don't want to see that get in the way of hybrid Intel/Nvidia support, which really does not need GLVND at all as long as users are okay with executing optirun. Eventually GLVND should make nvidia-headless-driver (as opposed to nvidia-driver) obsolete, but I think it is needed until then.
 
390.129 uses GLVND

Are you sure?

Next step from there would be to have the Nvidia driver port take responsibility for installing GLVND

430.40 already does that. It can't work any other way.

and without breaking Mesa

Somebody has to enable libglvnd in Mesa port. In fact, for now we can forget about nvidia-driver port and its libGL.so override habit. If Mesa is compiled with libglvnd support, it will work with that configuration anyway. It just means we would have two copies of every libglvnd binary installed (one from nvidia-driver and another from libglvnd port), which is a bit messy.
 
Are you sure?
No, more likely I got the 430.x parts confused with the 390.x parts of the patch.

Requirement of the actual libGL implementation to explicitly support GLVND (and inability of such an implementation to work without GLVND once it supports it) seem like serious design bugs in GLVND itself, but what can we do? There are other engineering decisions from Nvidia that I don't question, so if it's what they give us with their drivers, we do have to accept it in that context, just not everywhere.

430.40 already does that. It can't work any other way.
430.40 needs libglvnd somewhere; it doesn't have to be /usr/local/lib/libGL.so.1.

Somebody has to enable libglvnd in Mesa port. In fact, for now we can forget about nvidia-driver port and its libGL.so override habit. If Mesa is compiled with libglvnd support, it will work with that configuration anyway. It just means we would have two copies of every libglvnd binary installed (one from nvidia-driver and another from libglvnd port), which is a bit messy.
No, it can work without cooperation from Mesa port; VirtualGL has always made that possible (unless for some reason GLVND breaks VirtualGL, but it was working with GLVND AND VirtualGL in Linux compat, so I don't think that is a problem).

From the perspective of FreeBSD project, what does GLVND accomplish?
Ability to switch GL provider simply by specifying a different DISPLAY variable? How was needing optirun worse than that?
Ability for one program to use multiple GPUs with differing drivers? That is significantly more useful, but nothing much uses this (I've done this only in a research setting; it was pre-GLVND and required one procedure table per GL implementation, essentially a customized OpenGL loading library).
Ability for an application to be linked once against a system GL library, and then work on any implementation? This has already worked for a long time thanks to ELF.

My point is that GLVND is marginally useful but neither significantly helps nor hinders the work that is needed.
 
Back
Top