So. Guinea pig here.
I did the below steps. I have put a *** for the the steps I have changed.
Ollama runs but only with CPU. Card is rtx3060, Tried CUDA_VISIBLE_DEVICES=0 ollama run llama3.1:8b but still CPU only
I would like to start looking at doing llama-cpp instead of ollama but it...
...so existing binaries face zero missing library errors.
Which Packages Actually Need Reinstallation?
Kernel Modules (kmods): Drivers like nvidia-driver, drm-kmod (GPU drivers), or virtualbox-ose-kmod interact directly with internal kernel symbols (KABI). Kernel symbol changes between minor...
...wheel files which dont exist.
The way my freebsd-cuda works is by using the updated version of shkhln dummy-uvm.so
to intercept calls to nvidia-uvm which doesnt exist on Freebsd.
https://github.com/NapoleonWils0n/freebsd-cuda
I mount the Nvidia driver and linuxlator Rocky libraries into...
I did say that the project depended on your work with the dummy-uvm.so
in the videos, documentation and notes to get Nvidia working.
Without the dummy-uvm.so using Jails and Podman with Cuda wouldnt be possible.
The post was just a brief one line description of how the project worked...
...I understand it correctly llama-cpp is the engine in Ollama but I'm new to all this ai kerfuffling.
I think I saw that you have a thread about finding an x11 guinea pig and I will reply to that since my build-from-ports nvidia drivers forces me to use x11 on this old unbalanced hardware...
You can run Ollama with Cuda on Freebsd using a Rocky Linux Podman container
The Nvidia Driver and Linuxulator libraries are mounted from Freebsd into the Podman container,
and you can start and stop Ollama with a Makefile by running make to start Ollama and make clean to stop it...
Great topic for a thread, thanks cracauer@.
Here's some technical aspects for running AI models locally on unbalanced crap hardware with llama-cpp, some random AI models from huggingface and aider.
The crap Hardware:
CPU Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz ~ 2011
16 Gb DD3 RAM @ 1600 ~...
...installation of FreeBSD-15.1-RELEASE-amd64 I found the installer to be user friendly. However, getting a Mate Desktop up and running with an Nvidia graphics card was just "above my pay-grade".
Arch-Linux: Arch is an involved by hand installation, requiring your complete attention. Gentoo...
the issue that i have nvidia card which not listed in /etc/X11/xrdp/xorg.conf DRMAllowlist
freebsd is really bad with nvidia - got alot of headaches getting desktop to work at all with my card
I believe this is only needed if you want some sort of hardware acceleration. I don't need that, and don't install those drivers, and it is working well for basic access.
I didn't see anything in the xorg log that clued me in on a problem, but I might be missing something.
I have found that...
...but now i have another issue
i need now to edit to /etc/X11/xrdp/xorg.conf , for my device but i dont know what i need to do drm-kmod is for amd and intel not nvidia so i used (nvidia-drm-kmod)
but now how to find my render device and should i keep DRMAllowlist as is or edit it for nvidia
hello i am using freebsd on my computer (not headless device and using DE on it and installed display drivers - nvidia)
and i wish to be able to connect to it its DE remotely not just ssh but both tigervnc and xrdp refuses to work
uname -a
FreeBSD freebsd 15.1-RELEASE-p3 FreeBSD...
If you already have up-to-date ports tree (for the driver version) and /usr/src/sys/ 100% in sync with your running kernel.
My assumption on previous comment was that the only remaining problem is that you cannot fetch NVIDIA driver distfile.
...which is incomparable to more than 1 million lines of (I suppose) mostly blindly generated code.
This sounds familiar... Aha!
enabled-the-nvidia-gpu-passthrough-inside-a-linux-virtual-machine-via-qemu-accelerated-with-bhyve.102901/
EDIT: Sadly, I got it wrong. There is also an...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
