# Ollama with CUDA in a Devuan Podman container on FreeBSD
`devuan-ollama.sh` builds and runs a Podman container on a FreeBSD host. The container has a Devuan userland and runs Ollama on an NVIDIA GPU through CUDA. It also passes through X11 or Wayland and PulseAudio. The whole setup lives in one POSIX `sh` script: every host-side step, from package installation to building the NVIDIA kernel module, is performed by the script itself, so nothing has to be run by hand around it.
The work is modelled on NapoleonWils0n's [freebsd-cuda](https://github.com/NapoleonWils0n/freebsd-cuda), which does the same thing with a Rocky Linux base. That repository has no license file, so the script does not copy any of its files into its own distribution. It clones upstream at a pinned commit (`50f64e44fb5f9267df8f56cd730d5609e656d665`) and uses a few files from that local clone.
This document covers four things: how the container works, how it was built, the problems hit along the way and how each was solved, and what is still untested.
## Tested configuration
The configuration below is the only one validated end to end. Anything that differs from it should be treated as untested.
| Component | Version |
|---|---|
| GPU | NVIDIA GeForce RTX 2080 Ti (Turing, compute 7.5) |
| Host | FreeBSD 16.0-CURRENT kernel (`kern.osreldate` 1600001) with a FreeBSD 15 userland |
| NVIDIA driver | 595.84 (kernel module built by the script, Linux libraries in `/compat/linux`) |
| Container base | `docker.io/devuan/devuan:excalibur` (Devuan 6, glibc 2.41) |
| Ollama | v0.34.0 (official Linux amd64 release) |
| Session | X11 |
Ollama detects the GPU in the container. Its log reports `library=CUDA compute=7.5 description="NVIDIA GeForce RTX 2080 Ti" libdirs=ollama,cuda_v12` with 10.7 GiB of VRAM visible.
## Quick start
```sh
./devuan-ollama.sh all
```
`all` runs every stage in order: `check`, `prepare`, `build`, `up`, `run` and `verify`. The stages can also be run one at a time:
```sh
./devuan-ollama.sh check # installs/configures host prerequisites, then checks the host
./devuan-ollama.sh prepare # driver, NVIDIA libs, Dockerfiles, compose.yaml, .env, audio
./devuan-ollama.sh build # dummy-uvm.so, base image, Ollama image
./devuan-ollama.sh up # starts the container
./devuan-ollama.sh run # starts `ollama serve`, waits for the API
./devuan-ollama.sh verify # host vs container GPU diagnostics, API, Ollama GPU detection
./devuan-ollama.sh shell # interactive shell in the container
./devuan-ollama.sh down # stops and removes the container
```
Run the script as your desktop user, not as root. Privileged steps go through `doas` (set `PRIV=sudo` if you use sudo). To confirm the GPU is actually used, open `shell`, load a model with `ollama run <model>`, and check that `ollama ps` shows `100% GPU` in the PROCESSOR column.
## How the container works
### Podman on FreeBSD is already a jail
On FreeBSD, Podman runs containers through `ocijail`: each container is a FreeBSD jail. When the image is a Linux image, the jail executes Linux binaries through the Linuxulator (`linux64`). No separate jail or VM layer is needed. The "Devuan jail" is simply the Devuan container image running inside a jail with the Linux ABI.
### GPU access
The kernel side of the GPU is the host's native FreeBSD `nvidia.ko`. The container sees `/dev/nvidia0` and `/dev/nvidiactl` through devfs, and the Linuxulator presents them as Linux character devices (major 195, `nvidiactl` on minor 255). For these nodes to be visible inside the jail at all, devfs ruleset 4, the jail ruleset, must unhide `nvidia*`.
The user-space side is the Linux build of the NVIDIA libraries: `libcuda`, `libnvidia-ml`, `libnvidia-ptxjitcompiler` and the rest. On FreeBSD they are installed under `/compat/linux/usr/lib64`. Their version must match the loaded kernel module exactly.
FreeBSD's NVIDIA driver has no `/dev/nvidia-uvm` device, and the CUDA runtime expects one. The upstream project handles this with a small preload shim, `dummy-uvm.so`, built from `uvm_ioctl_override.c`. It intercepts the runtime's access to the UVM device. The shim is loaded with `LD_PRELOAD` inside the container.
The container also needs the Linuxulator pseudo-filesystems from the host: `linprocfs` as `/proc`, `linsysfs` as `/sys`, and `fdescfs` (mounted with `linrdlnk`) as `/dev/fd`.
### How the NVIDIA libraries reach Devuan
Upstream bind-mounts the host's entire `/compat/linux/usr/lib64` into the container and puts it on `LD_LIBRARY_PATH`. This works for upstream because the container is Rocky and so is the Linuxulator base on the host. That directory, however, holds a complete Rocky 9 userland, including `libc.so.6`. Mounted the same way into Devuan, it would put Rocky's glibc ahead of Devuan's own.
The script therefore copies only NVIDIA-owned files, and only those of the exact driver version, into `nvidia-host/`: `libcuda*`, `libnvcuvid*` and `lib*nvidia*`. It leaves out glibc and the glvnd front-ends, so `libGL.so.1`, `libEGL.so.1` and the like still come from Devuan. It rewrites absolute symlinks as relative ones and uses `readelf` to create any missing SONAME links, which matters for hand-built driver installs. The directory is mounted read-only at `/usr/lib/nvidia-host`, and `LD_LIBRARY_PATH` points only there. `nvidia-smi` is mounted separately from `/compat/linux/usr/bin`.
`NVLIB_MODE=bake` copies the libraries into the image at build time instead of mounting them.
### Display and audio
For X11, the host's `/tmp` is mounted read-only, which makes `/tmp/.X11-unix/X0` reachable. For Wayland, `XDG_RUNTIME_DIR` is mounted read-only and the compositor socket is symlinked into a writable runtime directory inside the container.
On the host, PulseAudio exposes a unix socket at `/tmp/pulseaudio.socket`, and the container uses it through `PULSE_SERVER`. The PulseAudio auth cookie is mounted read-only at `~/.pulse-cookie` and referenced through `PULSE_COOKIE`. Ollama itself uses neither display nor audio. They are there because the base image is meant to serve GUI applications as well.
### Ollama
The Ollama image downloads the official `ollama-linux-amd64.tar.zst` release. Download, extraction and deletion of the roughly 1.4 GB tarball happen in a single `RUN`, so the tarball never stays in an image layer. `start-ollama`, taken from upstream, selects the `cuda_v12` backend on Turing GPUs.
The container uses host networking, so the API answers on `127.0.0.1:11434`. Models are stored on the host in `~/.ollama`, and the server log goes to `~/.ollama/serve.log`.
### The kernel module: why the script builds it
The host this was developed on runs a 16-CURRENT kernel on a 15 userland. The packaged `nvidia.ko` targets 15 and declares a dependency on a kernel version in the 15 range, so the 16 kernel refuses it. Faking the kernel version would not be a fix. The kernel binary interface (structure layouts, internal function signatures) differs between major versions, and a mismatched module would panic or silently corrupt memory. The only correct fix is to compile the module against the source of the running kernel.
`prepare` does this whenever the driver is not attached. It takes the driver version from the `libcuda.so.X.Y` in `/compat/linux/usr/lib64`, so module and libraries match by construction. Then it:
1. Locates the running kernel's source tree. It tries `KERNEL_SRC`, then the source path implied by the build path in `uname -v`, then the current directory, then `/usr/src`. A tree is accepted only if its `__FreeBSD_version` equals `kern.osreldate`.
2. Downloads NVIDIA's FreeBSD driver tarball for that version and checks it against the SHA-256 file NVIDIA publishes alongside it.
3. Applies the FreeBSD patches from the `x11/nvidia-kmod` port. The ports tree is a sparse clone of about 3.5 MB containing only the framework and the two NVIDIA ports. The port is used only for fetch, extract and patch.
4. Builds `src/nvidia` directly with `SYSDIR` set to the running kernel's `sys/`.
5. Caches the result as `kmod/nvidia-<version>-<osreldate>.ko` and loads it from there, with GSP firmware disabled (see problem 8 below).
Nothing is written to `/boot/modules` or `loader.conf`. After a reboot, `up` and `run` load the cached module again. This is a deliberate choice, for three reasons:
- **No package conflicts.** The packaged `nvidia.ko` stays untouched, so `pkg upgrade` cannot overwrite the module or break it.
- **Automatic rebuilds.** A kernel rebuild changes `osreldate`, and the script then compiles a new module instead of failing at boot.
- **Passthrough stays possible.** The GPU is claimed by `nvidia.ko` only when the container needs it, so it remains free for bhyve passthrough the rest of the time.
## How it was built: problems and fixes
The script was developed iteratively on real hardware. The problems below are listed in the order they appeared. Several were mistakes in the script, and they are documented because the same traps will catch other people.
### Design decisions taken before the first run
These came from comparing the upstream Rocky files against Devuan, not from observed failures.
- **Library isolation.** The whole-directory mount of `/compat/linux/usr/lib64` would put Rocky's glibc ahead of Devuan's. The script copies only NVIDIA libraries instead, as described above.
- **Groups.** In Devuan, GID 44 already exists as `video`, so upstream's `groupadd -g 44` would fail. The script creates a group only when its numeric GID is missing. Upstream's GIDs 10 and 39 are Rocky-specific and were dropped.
- **apt under the Linuxulator.** The script raises `APT::Cache-Start` and sets `DEBIAN_FRONTEND=noninteractive`. Starting from the official Devuan OCI image, rather than debootstrap, avoids debootstrap's second-stage problems under the Linuxulator entirely. In practice `apt-get` and `gcc` ran without trouble inside Devuan excalibur, glibc 2.41, under the Linuxulator, even though upstream's working setup uses glibc 2.34.
- **`dummy-uvm.so`.** The shim is compiled inside Devuan instead of reusing upstream's Rocky 9.3 binary. `UVM_SOURCE=prebuilt` remains as a fallback: the upstream binary needs at most `GLIBC_2.34` and runs on Devuan.
- **Image size.** The Ollama tarball is handled in one layer. Upstream downloads it in one layer and deletes it in the next, which leaves the 1.4 GB in the image.
### 1. "nvidia.ko not loaded", and a misleading `kldload`
The first `prepare` stopped because `hw.nvidia.version` did not exist. Two detection mistakes followed. The script first looked for the module by file name with `kldstat -n`. It then tried `kldload nvidia` and got `module already loaded or in kernel`, even though no NVIDIA driver was active.
The lesson: a bare name without `.ko` passed to `kldload` is resolved as a *module* name, and here it produced a false positive. Neither `kldstat` nor `hw.nvidia.version` turned out to be a dependable signal. The script now treats the presence of `/dev/nvidiactl` as the signal that the driver is attached, and loads modules by file name or absolute path only.
### 2. The packaged module does not load on a 16 kernel
`kldload nvidia.ko` still answered `module already loaded or in kernel`. The real cause appeared only in `dmesg`:
```
KLD nvidia.ko: depends on kernel - not available or version mismatch
```
In other words, the 15 package on a 16 kernel. The fix is the in-script module build described earlier. Two details matter:
- **The ports framework refuses mixed hosts by default.** `bsd.port.mk` reads `OSVERSION` from `/usr/include/sys/param.h`, which is 15, and compares its major version with `uname -r`, which is 16. The build stops on that mismatch unless `OSVERSION` is passed explicitly. The script passes the running kernel's version.
- **One warning during the build is harmless.** `Symbol nv_kthread_task_callback is not present in nvidia.ko.full` appears because the port's symbol-export patch targets 595.99.02 and that symbol does not exist in 595.84. It concerns `nvidia-drm` only and is irrelevant for CUDA.
### 3. `podman-compose` missing
The script gained a `deps` stage, and `check` now runs it first. It installs `podman-suite`, `podman-compose`, `git`, `curl` and `pulseaudio` by *origin*, for example `pkg install sysutils/podman-compose`. The package name carries the Python flavor, currently `py312-podman-compose` where upstream's README says `py311`, and installing by origin does not depend on it.
### 4. `/etc/defaults/devfs.rules` "can't open file"
`deps` builds the jail ruleset by copying the system's `devfsrules_jail` block and adding one rule, `add path 'nvidia*' unhide`. awk reported `can't open file /etc/defaults/devfs.rules`, although the file exists. FreeBSD installs it with mode 600, and awk does not distinguish "missing" from "permission denied". Both devfs rules files are now read with privileges.
The script deliberately does not use upstream's variant, which redefines ruleset 4 without `$devfsrules_hide_all` and so makes all of `/dev` visible inside every jail. It also refuses to edit an `/etc/devfs.rules` that already defines ruleset 4. Keep in mind that ruleset 4 applies to all jails, so other jails will see the NVIDIA devices too.
### 5. Linux ABI loaded, but its filesystems not mounted
`check` reported the Linux ABI as loaded while `/compat/linux/proc`, `sys` and `dev/fd` were not mounted. Most likely `linux64` had been pulled in as a dependency of `nvidia.ko`, so `rc.d/linux` never ran. Running `service linux onestart` then did not create the mounts on this system either. The cause was not identified; `linux_mounts_enable="NO"` in `rc.conf` is a plausible one.
The script no longer relies on `rc.d/linux` for this. It mounts `linprocfs`, `linsysfs`, `devfs` and `fdescfs` (with `linrdlnk`) itself, in order and only where missing, under the real path of `compat.linux.emul_path`.
### 6. `dummy-uvm.so: Permission denied`
The shim is compiled inside a container, where it is written as root into a host directory. The next step, running as the user on the host, could not read it. The file is now given mode 0755 inside the container and handed back to the user with `chown` on the host.
### 7. `run` did nothing and printed nothing
Upstream's `wrapper-podman` starts the application by name through `PATH` and sends all its output to `/dev/null`. Rocky's default `.bashrc` adds `~/bin` to `PATH`. Debian's, and therefore Devuan's, does not. `start-ollama` was never found, and the error vanished with the discarded output.
`run` no longer uses the upstream wrapper. It starts `~/bin/start-ollama` by full path, sets `LD_LIBRARY_PATH` and `LD_PRELOAD` explicitly, detaches it with `setsid`, and logs to `~/.ollama/serve.log`. It then waits up to 30 seconds for the API. It also checks through `sockstat` and `jls` that whatever listens on port 11434 belongs to the container's jail and not to some other process.
### 8. "No devices found" on the host and in the container
`nvidia-smi -L` reported `No devices found.` both inside the container and on the host with the Linux binary, although `/dev/nvidia0` existed. Since the host failed as well, the problem sat between the module and the GPU, not in the container. `dmesg` showed:
```
firmware_get_flags: insufficient privileges to load firmware image nvidia_gsp_tu10x_fw
NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x61:0x56:2726)
```
The driver had GSP firmware enabled, `hw.nvidia.registry.EnableGpuFirmware: 1`. The firmware load is triggered on the first open of the device, and a regular user's process is not allowed to load kernel modules, so GPU initialization failed.
The port disables GSP by default, because of suspend/resume issues, and pre-RTX-50 GPUs work without it. In this build the default ended up enabled anyway. The registry is read only at module load, so the value has to be set before `kldload`, exactly like a `loader.conf` tunable. The script now runs `kenv hw.nvidia.registry.EnableGpuFirmware=0` before every load. The value is configurable through `NVIDIA_GSP`.
The script also changed its notion of a working driver. It no longer accepts "`/dev/nvidiactl` exists", only "`nvidia-smi -L` lists a GPU". When the driver is attached but the GPU is not initialized, the script reloads the module automatically.
### 9. The reload path: exec sessions and `kldunload`
The first automatic reload hit two more problems.
- **Podman refused to remove the container.** `podman-compose down` failed with `has active exec sessions, refusing to clean up`, a leftover of the old wrapper's session. `podman rm -f` terminates exec sessions, so the script now uses it before `compose down`.
- **FreeBSD's `kldunload` has no `-m`.** Unlike `kldstat`, it accepts only a file name or an id. A first attempt to find the id through a module named `nvidia` in `kldstat -v` found nothing on this system. The script now identifies the kld by file name, either `nvidia.ko` or its own `nvidia-<ver>-<osrel>.ko`, falls back to the module names, and unloads with `kldunload -i`. It unloads `nvidia-modeset.ko` and `nvidia-drm.ko` first when they are present.
After this, `nvidia-smi` listed the RTX 2080 Ti on the host and in the container, and Ollama reported `library=CUDA` with the GPU's full VRAM.
### 10. Audio
Upstream's approach needs `pavucontrol` kept open so that PulseAudio does not exit when idle, and it never passes the auth cookie to the container. The script now works as follows:
- **Daemon.** It starts PulseAudio with `--exit-idle-time=-1`, so no helper window is needed.
- **Socket.** If a daemon is already running without the socket, as can happen with pipewire-pulse or a different configuration, the script adds it at runtime with `pactl load-module`. A stale socket left by a dead daemon is removed first.
- **Configuration.** It writes a minimal `~/.config/pulse/default.pa` only if none exists.
- **Cookie.** It mounts the cookie read-only at `~/.pulse-cookie` in the container. The mount point is deliberately outside `~/.config`, because the container runtime would otherwise create that directory as root.
Audio from the host to the container has been checked at the socket level (`pactl` answers on the socket). It has not yet been tested with an application inside the container.
## Configuration
All settings are environment variables:
| Variable | Default | Purpose |
|---|---|---|
| `DEVUAN_RELEASE` | `excalibur` | Devuan image tag |
| `PROJECT_DIR` | `~/podman/freebsd-devuan` | Working directory for everything the script generates |
| `NVIDIA_LIB_SRC` | `/compat/linux/usr/lib64` | Where the Linux NVIDIA libraries live |
| `NVIDIA_SMI_SRC` | `/compat/linux/usr/bin/nvidia-smi` | Linux `nvidia-smi` |
| `NVIDIA_DRIVER_VERSION` | from `libcuda.so.X.Y` | Force the driver version |
| `NVIDIA_KMOD` | empty (build) | Path to a ready-made `nvidia.ko` instead of building one |
| `KERNEL_SRC` | auto-detected | Source tree of the running kernel |
| `NVIDIA_GSP` | `0` | `EnableGpuFirmware` at module load |
| `NVLIB_MODE` | `mount` | `mount` or `bake` the NVIDIA libraries |
| `UVM_SOURCE` | `build` | `build` the shim in Devuan, or use upstream's `prebuilt` one |
| `OLLAMA_VERSION` | `latest` | Ollama release tag |
| `OLLAMA_DATA` | `~/.ollama` | Models and server log |
| `CONTAINER_NAME` | `devuan-ollama` | Container name |
| `PRIV` | `doas` | Privilege escalation command |
| `SESSION` | autodetected | `x11` or `wayland` |
## What the script changes on the host
Everything it does is idempotent. On the host, the script:
- installs the missing packages;
- adds `fdescfs` on `/dev/fd` to `/etc/fstab`;
- appends a jail ruleset to `/etc/devfs.rules`, after writing a timestamped backup, and restarts devfs;
- mounts the Linuxulator filesystems;
- starts PulseAudio for your user;
- creates `~/.config/pulse/default.pa` if it does not exist;
- loads its own `nvidia.ko` and sets one `kenv` tunable, neither of which persists across a reboot.
Everything else stays inside `PROJECT_DIR` and `~/.ollama`.
## Limitations and untested areas
- **Wayland.** The Wayland path has not been tested. The reference machine runs X11.
- **GUI OpenGL/Vulkan in the container.** Not covered. The copied library set is compute-oriented: it omits `libnvidia-egl-wayland/gbm/xcb`, which are versioned independently, and the EGL/Vulkan vendor JSON files.
- **Kernel compatibility.** The module build depends on the NVIDIA version compiling against your kernel's internal interfaces. On a CURRENT kernel an internal API change can break it, and there is no shortcut when that happens.
- **Port patches.** They are taken from the current ports tree, which targets a newer driver point release, and applied to your version. They applied cleanly to 595.84 but are not guaranteed to for every version.
- **GSP.** RTX 50-series GPUs are known to need GSP firmware. The script does not build the GSP firmware modules, so `NVIDIA_GSP=1` would reproduce problem 8 on those cards.
- **GPU exclusivity.** A GPU bound to `ppt(4)` for bhyve passthrough cannot be used by the container at the same time. `check` warns about this.
- **Security.** Ruleset 4 is shared by all jails, and the container user is a member of the host's `wheel` and `video` GIDs, as in upstream.
## Credits
The architecture, `start-ollama`, and the `dummy-uvm` source come from NapoleonWils0n's [freebsd-cuda](https://github.com/NapoleonWils0n/freebsd-cuda). The FreeBSD NVIDIA patches come from the FreeBSD ports tree.
The Devuan port, the in-script kernel module build, and the fixes described above were developed by Marietto with an AI assistant (Claude, by Anthropic). Every step was validated on the hardware listed at the top.
