Technical aspects of running local LLMs on FreeBSD

There is no FreeBSD-specific CUDA code, obviously.
Click to expand...

we can agree on that
makes a change

As you said ports have to be patched

And there are some ports that havent been patched like Handbrake or Blender,
because you only have so much time, and as you said about Handbrake

FYI, there is no NVENC with Handbrake, because I didn't bother to submit the corresponding patches to the port.
Click to expand...

That's correct isnt it.

So the project is addressing 2 issues

1) Ports that havent been patched to work with Cuda,
and installing the Linux version with Podman to get Cuda working.

2) Linux applications that require Cuda but cant be ported to Freebsd like Davinci Resolve because of licensing issues.

So if a port hasnt been patched the user has an alternative,
and can install the Linux version of the application with Podman and get it working with Cuda.

Rather than asking you or other devs to patch lots of different ports,
which you might not have the time for.

Surely thats a good thing from your point of view,
as it means less work patching.

Having pre configured applications that are easy to deploy,
and making things easier for user and giving them more choice is a good thing.

The end goal is getting an application working with Cuda for users,
we just have different ways of going about it.
 
ThisIsAGoodThread.png
 
To run a application/port that has been patched you have to prefix the command with nv-sglrun

for example to use ffmpeg with nvenc

Code: 
nv-sglrun ffmpeg

script example

Code: 
#!/bin/sh

#===============================================================================
# convert video to h265/aac
#===============================================================================


#===============================================================================
# script usage
#===============================================================================

usage()
{
# if argument passed to function echo it
[ -z "${1}" ] || echo "! ${1}"
# display help
echo "\
# convert video to h265/aac

$(basename "$0") -i input.mov -o output.mp4
-i infile.mov
-o outfile.mov :optional agument # if option not provided defaults to input-name.mp4"
exit 2
}


#===============================================================================
# error messages
#===============================================================================

NOTFILE_ERR='not a file'
INVALID_OPT_ERR='Invalid option:'
REQ_ARG_ERR='requires an argument'
WRONG_ARGS_ERR='wrong number of arguments passed to script'


#===============================================================================
# check number of aruments passed to script
#===============================================================================

[ $# -gt 0 ] || usage "${WRONG_ARGS_ERR}"


#===============================================================================
# getopts check options passed to script
#===============================================================================

while getopts ':i:o:h' opt
do
  case ${opt} in
     i) input="${OPTARG}"
    [ -f "${input}" ] || usage "${input} ${NOTFILE_ERR}";;
     o) output="${OPTARG}";;
     h) usage;;
     \?) usage "${INVALID_OPT_ERR} ${OPTARG}" 1>&2;;
     :) usage "${INVALID_OPT_ERR} ${OPTARG} ${REQ_ARG_ERR}" 1>&2;;
  esac
done
shift $((OPTIND-1))


#===============================================================================
# variables
#===============================================================================

input_nopath="${input##*/}"
input_name="${input_nopath%.*}"

# defaults for variables if not defined
output_default="${input_name}.mp4"


#===============================================================================
# functions
#===============================================================================

# h265 function
h265 () {
    nv-sglrun \
    ffmpeg \
    -hide_banner \
    -stats -v panic \
    -i "${input}" \
    -c:v hevc_nvenc \
    -pix_fmt p010le \
    -preset slow \
    -tier high \
    -rc vbr \
    -cq 22 \
    -b:v 0 \
    -maxrate 50M \
    -c:a aac \
    -b:a 320k \
    -ar 48000 \
    "${output:=${output_default}}"
}

# run the h265 function
h265 "${input}"

for gui applications you need to modify the desktop entry

for example to get obs studio working with nvenc

Code: 
[i] Yes Master ? ls -l ~/.local/share/applications/com.obsproject.Studio.desktop
-rw-r--r--  1 djwilcox djwilcox 395  6 Jul 15:04 /home/djwilcox/.local/share/applications/com.obsproject.Studio.desktop

com.obsproject.Studio.desktop

Code: 
[Desktop Entry]
Version=1.0
Name=OBS
GenericName=Streaming/Recording Software
Comment=Free and Open Source Streaming/Recording Software
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'
Icon=com.obsproject.Studio
Terminal=false
Type=Application
Categories=AudioVideo;Recorder;
StartupNotify=true
StartupWMClass=obs

notice the exec line

Code: 
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'

as opposed to just running obs

Code: 
Exec=obs --websocket_ipv4_only

However as i said if you try and install something like whisperx
which doesnt have a native Freebsd package with pip or conda with the Linuxulator or a Jail

Then it will fail to install because it will detect its running on Freebsd
and try and download Freebsd python wheels that dont exist

there are Freebsd python packages for torch

Code: 
[i] Yes Master ? pkg search torch
py312-facenet-pytorch-2.5.3_4  Pretrained PyTorch face detection and recognition models
py312-lion-pytorch-0.2.4       PyTorch: Lion optimizer
py312-pytorch-2.12.1           PyTorch: Tensors and dynamic neural networks in Python
py312-pytorch-lightning-2.6.5  Lightweight PyTorch wrapper for ML researchers
py312-pytorchvideo-0.1.5_4     Video understanding deep learning library
py312-torch-geometric-2.8.0    Graph neural network library for PyTorch
py312-torchao-0.17.0           PyTorch: Package for applying ao techniques to GPU models
py312-torchaudio-2.11.0        PyTorch-based audio signal processing and machine learning library
py312-torchcodec-0.13.0        PyTorch media decoding and encoding
py312-torchdata-0.11.0         PyTorch: Composable data loading modules for PyTorch
py312-torchmetrics-1.9.0       PyTorch native metrics
py312-torchsde-0.2.6_2         SDE solvers and stochastic adjoint sensitivity analysis in PyTorch
py312-torchsummary-1.5.1_2     PyTorch: Model summary in PyTorch
py312-torchvision-0.27.0       PyTorch: Datasets, transforms and models specific to computer vision
pytorch-2.12.1                 Tensors and dynamic neural networks in Python (C++ library)
[

But they may not be the correct version for python application you are trying to install
and the application may require additional python libraries that dont have a Freebsd package

So the advantage of using Podman is you can set the container platform and os to Linux
which presents a Linux environment to applications like python so they download the python linux wheels

So for native Freebsd packages that have been patched to support nvenc for example
you can prefix the command with nv-sglrun or modify the desktop entry
 
BSD Jedi has some Ollama tutorial and a new one with a plug for some budget option cloud models.

I run 16GB vRAM nvidia on Debian Linux Ollama - rarely used, mainly as brain for Home Assistant....for fun.
I have good experience with Mixture of Experts models - gemma4. They really seem to use just parts of the model they actually need for given task.

Alas at work I have some good Ryzen AMD CPU, no vRAM to speak of but 64GB of DDR5 RAM (bought a year ago, yey) and it is surprisingly capable of running much bigger models than my home machine. Slow but who cares when crunching config files and not having conversations.

I also bought Google Coral TPUs on cheap but there is no chance I can make them work on naked FreeBSD - python libraries expect Linux. I was thinking of making a robot car with visual recognition on FreeBSD but all the python libraries are just made for Linux Rapsberry "drivers" too.

For more serious and private work 128GB is a must. But as OP pointed out if privacy is not a must, cloud models are better investment. Or rather 4,000 USD machine which is way dumber is just really bad investment right now.
 
You must log in or register to reply here.
Back
Top