I have managed to get Ollama 24 working with Cuda and Podman
Using my new project freebsd-cuda ( which im still working on )
I have 4 Podman containers in the project
1) Ollama working with Cuda
The models are stored on the Freebsd side in ~/.ollama
You enter the podman container and run ollama serve
And then on the Freebsd you can connect to Ollama running in the container as if it was running natively
Either using the terminal or a front end application for Ollama
2) ffmpeg with nvenc encoding
bin directory for scripts mounted from Freebsd into the container
video directory mounted from Freebsd into the container so you can process files stored on the Freebsd side
3) Firefox with Widevine drm and hardware accelerated video
Persistent storage that retains all your Firefox setting
and download directory from Freebsd mounted in the Podman container
Wayland and Pulseaudio socket for the window and audio
Its actually using pipewire-pulse but using the pulseaudio.socket
Firefox launched with a desktop entry on Freebsd which starts the Podman container if its not running
and then opens Firefox like a native application
wrapper scripts for Freebsd and Podman that work with desktop entries
to launch applications in Podman container
4) WhisperX for audio transcription working with Cuda
video directory mounted from Freebsd into the container so you can process files stored on the Freebsd side
Podman containers are actually Jails
The reason this is important is that you can run Jails commands on the Podman containers
You cant launch applications from the Freebsd side using doas podman exec
because it will run the application as root in the Jail and change all the permissions
So instead what you do is use jexec
In this snippet from the wrapper-freebsd script we pass in the USER from Freebsd
the Podman jail id and the app to launch
By using jexec it launches the application in the container as a non root user
which is the same users as on the Freebsd side
To do:
I need to create an X11 version of the Podman containers
The last time i checked the X11 socket was stored in /tmp
Using my new project freebsd-cuda ( which im still working on )
I have 4 Podman containers in the project
1) Ollama working with Cuda
The models are stored on the Freebsd side in ~/.ollama
You enter the podman container and run ollama serve
And then on the Freebsd you can connect to Ollama running in the container as if it was running natively
Either using the terminal or a front end application for Ollama
2) ffmpeg with nvenc encoding
bin directory for scripts mounted from Freebsd into the container
video directory mounted from Freebsd into the container so you can process files stored on the Freebsd side
3) Firefox with Widevine drm and hardware accelerated video
Persistent storage that retains all your Firefox setting
and download directory from Freebsd mounted in the Podman container
Wayland and Pulseaudio socket for the window and audio
Its actually using pipewire-pulse but using the pulseaudio.socket
Firefox launched with a desktop entry on Freebsd which starts the Podman container if its not running
and then opens Firefox like a native application
wrapper scripts for Freebsd and Podman that work with desktop entries
to launch applications in Podman container
4) WhisperX for audio transcription working with Cuda
video directory mounted from Freebsd into the container so you can process files stored on the Freebsd side
Podman containers are actually Jails
The reason this is important is that you can run Jails commands on the Podman containers
You cant launch applications from the Freebsd side using doas podman exec
because it will run the application as root in the Jail and change all the permissions
So instead what you do is use jexec
In this snippet from the wrapper-freebsd script we pass in the USER from Freebsd
the Podman jail id and the app to launch
Code:
doas jexec -u "${USER}" "${jail_id}" /bin/bash -ilc "${HOME}/bin/wrapper-podman -a ${app}" -- "${@}"By using jexec it launches the application in the container as a non root user
which is the same users as on the Freebsd side
To do:
I need to create an X11 version of the Podman containers
The last time i checked the X11 socket was stored in /tmp