general/other Using Linux Emuation as a means to utilize CUDA.

O

OSExplorer

I was tasked with seeing the viability of FreeBSD as a workspace as we need to shift away from CentOS. FreeBSD obviously does not have official support for CUDA drivers, however I was told it "could" make use of CentOS emulation which I was asked to try. If it is completely impossible to accomplish this, please let me know and disregard the rest of this post.

I first installed gcc and the basic nvidia display drivers with pkg install gcc and pkg install nvidia-driver respectively. I then used ports to install emulators/linux_base-c7 and textproc/linux-c7-libxml2 using make install distclean.

Once this setup was complete I fetched the latest CentOS 7 local installer for CUDA (11.2.2). As root I ran the installer and using the default options in the commandline user interface chose to install. This left me with a driver installation failure error which I found in /var/nvidia-installer.log stating I needed modprobe which could be obtained via kmod or module-init-tools. I attempted to resolve this with pkg install drm-kmod but with no success.

From there I have reached a bit of a sticking point. If this isn't a completely hopeless endeavor I would appreciate advice.
 
Short answer: Linux compatibility is for the userland (for running Linux applications), not inside the kernel. So, you can't use a kernel driver/module from Linux.

The drivers in graphics/drm-kmod actually ARE drivers from Linux, but a) rebuilt for FreeBSD and b) include an in-kernel compatibility layer "linuxkpi" which is "just enough" for these drivers. So, even if you would get Nvidia's source for Linux, it would still be extremely unlikely it would work with "linuxkpi".
 
Well, what would you expect from a proprietary programming interface only usable with Nvidia chips?

Not that the basic idea behind CUDA would be that bad, why NOT use the power of a modern GPU for things it's optimized for, even when not rendering graphics? But then, doesn't OpenCL fit the bill here, as an open standard NOT tied to Nvidia? Well ok, probably no support with binary Nvidia drivers on FreeBSD either ;)
 
CUDA depends on a few missing driver pieces: nv_register_user_pages/nv_unregister_user_pages/os_lock_user_pages/os_unlock_user_pages in nvidia.ko and the whole nvidia-uvm.ko kernel module. Those missing functions reside in the OS shim part of the driver, so it's possible to copy them from the Linux driver, see a proof-of-concept patch here: https://github.com/shkhln/revird-aidivn/commit/a68f5a4abe767af9fda80aae3851b59fea1b6277. (Note that this patch exists purely for demo purposes, I have no intention of doing anything useful with it.) UVM is entirely open-source, it even has a permissive license, however it's quite large, so to port that thing somebody would need to throw some real $ at a competent kernel developer. (Not at me, mind you.) UVM is technically optional, so some programs work happily without it and some likely won't. To test your applications in Linuxulator without UVM use this little hack: https://gist.github.com/shkhln/40ef290463e78fb2b0000c60f4ad797e.
 
OpenCL is an open standard, implementations are a different issue, but at least some open-source implementations exist. CUDA is tied to Nvidia and nothing else. So, I'd say that's indeed a relevant difference.
 
Modprobe is a Linux specific tool. The FreeBSD counterpart is kldload.
You may need to massage the installer a bit, which most likely is a shell script.
Click to expand...
This is what I hoped.

Short answer: Linux compatibility is for the userland (for running Linux applications), not inside the kernel. So, you can't use a kernel driver/module from Linux.

The drivers in graphics/drm-kmod actually ARE drivers from Linux, but a) rebuilt for FreeBSD and b) include an in-kernel compatibility layer "linuxkpi" which is "just enough" for these drivers. So, even if you would get Nvidia's source for Linux, it would still be extremely unlikely it would work with "linuxkpi".
Click to expand...
But this is what I feared.

CUDA depends on a few missing driver pieces: nv_register_user_pages/nv_unregister_user_pages/os_lock_user_pages/os_unlock_user_pages in nvidia.ko and the whole nvidia-uvm.ko kernel module. Those missing functions reside in the OS shim part of the driver, so it's possible to copy them from the Linux driver, see a proof-of-concept patch here: https://github.com/shkhln/revird-aidivn/commit/a68f5a4abe767af9fda80aae3851b59fea1b6277. (Note that this patch exists purely for demo purposes, I have no intention of doing anything useful with it.) UVM is entirely open-source, it even has a permissive license, however it's quite large, so to port that thing somebody would need to throw some real $ at a competent kernel developer. (Not at me, mind you.) UVM is technically optional, so some programs work happily without it and some likely won't. To test your applications in Linuxulator without UVM use this little hack: https://gist.github.com/shkhln/40ef290463e78fb2b0000c60f4ad797e.
Click to expand...
Thank you though for this. I will see if I can hack anything out with it before throwing in the towel, so to speak.
 
Thanks this explains a lot of questions for me with regards for Nvidia GPU on using their CUDA toolkit on FreeBSD. Seems like it could be done after some kernel, driver patching/porting for FreeBSD. I might as well just save money before I buy a Nvidia GPU so I do not need to get into this issue, might as well learn kernel and driver programming for FreeBSD so I can port the CUDA Linux tookit on FreeBSD.

Is this even more complicated for AMD GPU tool kits, like for ROCm or AMD-APP-SDK-v3.0 which works officially on Linux ?

Thanks.
 
You must log in or register to reply here.
Back
Top