This is assuming you already have a working graphics card driver with Vulkan. You can test with `vulkaninfo --summary` from pkg vulkan-tools. Working Vulkan is in the binary NVidia drivers and the AMD drivers (including CPU-integrated GPU). Dunno about Intel GPUs.
This will install the llama.cpp LLM runner, which is an intermediate difficulty...
I wanted to waffle a bit more about the hardware involved. You basically have two approaches:
Get a fat GPU with a certain amount of VRAM. You will be able to run LLMs up to the VRAM size fast. Overflowing models up to RAM size of the machine will run slow but not necessarily catastrophic if you still have a good chunk in VRAM.
Get a machine with integrated GPU that shares RAM with the GPU. This gets you much more memory for the GPU than you can afford with a dedicated GPU. It will run models up to RAM size at mediocre to OK speed.
Cost-wise it is a wash. For $3300-$3500 you can pick between:
A used NVidia 5090 with 32 GB VRAM
3x NVidia 3090 and a really fat power supply
An AMD Strix Halo (Ryzen AI 395) with 128 GB RAM (shared GPU memory)
An Apple Mac with a M4 Max and 128 GB RAM (shared GPU memory)
The Apple Mac doesn't run all models well. Some models are marked as optimized for Metal (Apple's equivalent to Vulkan).
I have been told that Strix Halo prefers to run Mixture of Expert models and doesn't do so well on dense models. I don't have one of those suckers so I can't comment. There's also the aspect of getting FreeBSD to first run on the thing (we have recently seen a failure) and getting Vulkan up.
I have the single NVidia card, which runs out of the box for FreeBSD. But alas everything bigger than 32 GB is slow. There are many interesting models between 32 and 128 GB. But on the other hand, the models that do fit are very fast. You can also use the performance advantage to increase the context size (although that costs further VRAM). I also plan to do extensive experiments with post-training and agents. Speed is more important in that case than when just running a chat through the web browser.
I do not know whether multi-GPU works for one of the llamas on FreeBSD. On paper 3x 3090 looks really attractive since it gives you 72 GB of VRAM for the same price. And functions as heating in winter. Just for starters it might be that multi-GPU in llama.cpp only works with CUDA, not Vulkan.
Theoretically the NVidia GPUs have another advantage: you can run all that GPU software that only has CUDA backends. At the time of this writing CUDA does not work on FreeBSD through Linuxulator, though.
Finally just a word on commercial LLMs: cost-wise it is clearly best to just use up as many of the $20/month plans. They are heavily sponsored. Buying your own hardware can't compete price-wise. But Anthropic might or might not have kicked Claude Code out of that plan:
Executive Summary: * In the later afternoon of April 21 2026, Anthropic removed access to Claude Code for its $20-a-month "Pro" Plans on various pricing pages. * Current Pro users appeared to still have access via the Claude web app. * Claude Code support documents, for a brief period...
Did you install llama and ggml from packages? Maybe this is an issue of being compiled in a context where native CPU was detected during compilation? Although Raptor Lake should not miss much except avx512.
Did you install llama and ggml from packages? Maybe this is an issue of being compiled in a context where native CPU was detected during compilation? Although Raptor Lake should not miss much except avx512.
That's possible. I'm not sure about the origin of everything.
It's also not a GENERIC 15 kernel. Some things are changed.I don't think it can be related but it may be a problem. System was still copying so I didn't want to reboot. I'll try a clean kernel and world in a few hours.
Qwen Studio offers comprehensive functionality spanning chatbot, image and video understanding, image generation, document processing, web search integration, tool utilization, and artifacts.
qwen.ai
This particular homepage is not very good. It doesn't have recommendations for parameters like temperature, kv cache and so on.
Heads up: version 8895 of llama.cpp has been committed to ports.
This changes the location where llama.cpp stores models automatically downloaded from huggingface (as in post #1 here).
Old cache: ~/.cache/llama.cpp/
New cache: ~/.cache/huggingface/hub
It will automatically move already downloaded models to the new location. If you have different filesystems under there you might be in for a surprise.
In my case the homedir is on NFS and there is a symlink for the old cache to a local NVMe SSD. So it moved things to NFS and afterwards I made to move them back via a new symlink to the SSD. Took 3 hours I'm never getting back.
On current processors, the performance degradation of scheduling avx512 code on an efficiency core instead of a performance core is significantly larger then the degradation of scheduling avx2 code on an e-core instead of a p-core, or in other words, the relative performance to p-cores of e-cores is higher when executing avx2 then it is when execuring avx512. Since FreeBSD-15.0 has no awareness of the cpu design, it will run the inference software on whatever cores are least loaded at the time the inference software is launched, and then will keep it sticked on that cores unless other cores have at least kern.sched.steal_tresh less load on them, which means odds are high that llama.cpp starts on an e-core, allocates mostly e-cores and then stays there. The CPU firmware will then identify llama to be avx heavier then most other workloads and generate an advisory asking the OS to reschedule it onto a p-core, however FreeBSD-15.0 will ignore that advisory and keep it on the e-core(s) nevertheless. So until FreeBSD learns how to address scheduler advisories (currently tracked under D54674 for scheduler advisories in general and D44456 for intel specific Intel Thread Director register processing) there's no point in using avx512 even if your CPU does support it.
Here's some technical aspects for running AI models locally on unbalanced crap hardware with llama-cpp, some random AI models from huggingface and aider.
The crap Hardware:
CPU Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz ~ 2011
aider - this one is a bit trickier - git clone the source and then pip install in a venv and fix the dependency errors - it can be done but requires some kerfuffling
Task for AI to do - the prompt, the testcase, whatever you want to call it. The goal is to make AI write the code and test until all errors are gone without intervention.
Create a new C file with a descriptive name. Inside it, implement a function to read the value for the CPU temperature using sysctlbyname with the modern path "dev.cpu.0.temperature". Note that FreeBSD returns this value as an integer in deci-Kelvin (e.g., 3000 means 300.0 Kelvin). Convert this value to Celsius by dividing by 10.0 and subtracting 273.15, then print it out in main() Also in the same code read the values for load average and print those out. If the compilation fails, intercept the errors and fix them.
success for temperature fails slightly for load averages (2/3)
~ 1 t/s
Warning:
If you run the below scripts you are responsible. You will see llama-cpp warnings like below so make sure you are safe. srv llama_server: -----------------
srv llama_server: CORS is set to allow all origins ('*') and no API key is set
srv llama_server: this can be a security risk (cross-origin attacks)
srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
srv llama_server: -----------------
srv llama_server: -----------------
srv llama_server: the following feature(s) are enabled:
srv llama_server: server tools (experimental)
srv llama_server: do not expose the server to untrusted environments
srv llama_server: -----------------
Comments:
- llama-cpp was installed from ports first, that might have helped to build it from source
- for aider the dependencies that caused errors were installed as pkg and then edited out from requirements.txt
- the reason speed drops for models sized > NVRAM is that the CPU/RAM slows everything down when offloading from NVRAM is being done
- there is margin in the memory management in llama startup script to allow kde and chromium to run at the same time as the ai model
- there is a feeling that Vulkan not always is giving memory back after stopping llama-server with ctrl-c
Script to start llama and pick a model:
sh:
#!/bin/sh
MODELPATH="$HOME/AI-models"
TOTAL_VRAM=12.0 # Physical VRAM in GB
MAX_LOCKED_RAM=10.0 # FreeBSD limit for mlock in GB
# Path to your working Python 3.12 binary inside your venv
PY_BIN="$HOME/venvs/aider-env/bin/python3"
run_llama() {
FILE_NAME="$1"
FILE_SIZE_GB="$2"
FULL_PATH="$MODELPATH/$FILE_NAME"
echo "──────────────────────────────────────────────────────"
echo "📊 Analyzing $FILE_NAME ($FILE_SIZE_GB GB)..."
# Verify NVIDIA GPU and driver status cleanly without screen flickering
echo "🎮 Verifying NVIDIA GPU status..."
if [ ! -c /dev/nvidia0 ]; then
echo "❌ Error: NVIDIA device node (/dev/nvidia0) is missing!"
echo " Please ensure the nvidia-driver is loaded properly."
return 1
fi
# Verify that the kernel module is active via sysctl
NV_VERSION=$(sysctl -n hw.nvidia.version 2>/dev/null)
if [ -z "$NV_VERSION" ]; then
echo "❌ Error: NVIDIA kernel module is not active or responsive!"
return 1
fi
echo "✅ GPU Status: NVIDIA Graphics Driver ($NV_VERSION) is active and ready."
echo "🧠 Reading model built-in metadata live..."
# Architecture-independent raw binary header parser using native Python struct unpack
TRAINED_CTX=$("$PY_BIN" -c '
import sys, struct
try:
with open("'"$FULL_PATH"'", "rb") as f:
# Read the first 5MB where the GGUF metadata directory lives
header = f.read(5000000)
# Locate the context_length byte sequence
idx = header.find(b"context_length")
if idx != -1:
# GGUF keys are followed by a type tag (UINT32 = 6) and the value.
# We scan a 128-byte window immediately following the key index
window = header[idx:idx+128]
# Step through the window and unpack 4-byte little-endian integers (UINT32)
for i in range(len(window) - 4):
val = struct.unpack("<I", window[i:i+4])[0]
# Filter out numbers that match standard context limits (8k to 256k)
if val in [8192, 16384, 32768, 65536, 131072, 163840, 262144]:
print(val)
sys.exit(0)
except Exception:
pass
print("")
')
# Sanitize the output string in a safe FreeBSD manner
TRAINED_CTX=$(echo "$TRAINED_CTX" | tr -cd '0-9')
if [ -z "$TRAINED_CTX" ] || [ "$TRAINED_CTX" -lt 2048 ]; then
TRAINED_CTX=16384
echo "ℹ Could not read trained context length. Using safe fallback: $TRAINED_CTX"
else
echo "🧠 Model is trained for a max of: $TRAINED_CTX tokens."
fi
# Advanced calculation with adaptive cache scaling to guarantee VRAM stability
EVAL_OPTS=$(awk -v vram="$TOTAL_VRAM" -v ram="$MAX_LOCKED_RAM" -v size="$FILE_SIZE_GB" -v max_ctx="$TRAINED_CTX" '
BEGIN {
system_vram_overhead = 1.5;
available_vram = vram - system_vram_overhead;
if (size < available_vram) {
ngl = 99;
ram_fallback = 0;
b = 512; ub = 256;
leftover_vram = available_vram - size;
if (size > 6.5) {
calculated_ctx = int((leftover_vram / 0.08) * 1000);
type = "q4_0";
} else {
calculated_ctx = int((leftover_vram / 0.16) * 1000);
type = "q8_0";
}
if (calculated_ctx >= max_ctx) {
ctx = max_ctx;
} else if (calculated_ctx >= 16384) {
ctx = calculated_ctx;
if (ctx > 24576) ctx = 24576;
} else {
calculated_ctx_q4 = int((leftover_vram / 0.08) * 1000);
ctx = calculated_ctx_q4;
if (ctx > max_ctx) ctx = max_ctx;
type = "q4_0";
}
if (ctx < 8192) ctx = 8192;
if (ctx > max_ctx) ctx = max_ctx;
ctx = int(ctx / 1024) * 1024;
if (ctx < 1024) ctx = 1024;
} else {
ctx = 16384;
type = "q4_0";
if (ctx > max_ctx) ctx = max_ctx;
kv_overhead = 1.5;
usable_vram_for_layers = available_vram - kv_overhead;
if (usable_vram_for_layers < 0) usable_vram_for_layers = 0;
vram_ratio = usable_vram_for_layers / size;
approx_total_layers = 64;
ngl = int(approx_total_layers * vram_ratio);
if (ngl < 0) ngl = 0;
if (ngl > 80) ngl = 80;
ram_fallback = size - usable_vram_for_layers;
b = 128; ub = 64;
}
if (ram_fallback > ram) {
print "ERROR: Model requires more mlock RAM than your allowed 10 GB!" > "/dev/stderr";
}
printf "%d %s %d %d %d\n", ctx, type, b, ub, ngl;
}')
read -r ctx type b ub ngl <<EOF
$EVAL_OPTS
EOF
if [ -z "$ctx" ]; then
echo "❌ Invalid parameters calculated. Aborting."
return 1
fi
echo "🎯 Optimal configuration calculated for your hardware:"
echo " -> Context Window (-c): $ctx tokens (Model max: $TRAINED_CTX)"
echo " -> Cache Memory Type: $type"
echo " -> GPU Layer Offload (-ngl): $ngl (99 = All layers on GPU)"
echo " -> Flash Attention: ON"
echo " -> Batch Allocation (-b / -ub): $b / $ub"
echo "──────────────────────────────────────────────────────"
echo "🚀 Initializing llama serve..."
llama-server \
-t 3 \
-b "$b" \
-ub "$ub" \
-c "$ctx" \
-np 1 \
-fa on \
--load-mode mlock \
--cache-ram 0 \
--cors-origins "*" \
--jinja \
--host 0.0.0.0 \
--port 8080 \
--tools all \
--alias kAI \
--cache-type-k "$type" \
--cache-type-v "$type" \
--temp 0.2 \
--top-p 0.1 \
--min-p 0.05 \
-m "$FULL_PATH"
}
# Check if directory exists
if [ ! -d "$MODELPATH" ]; then
echo "❌ Error: Model directory $MODELPATH does not exist."
exit 1
fi
echo "Pick an AI model to run: "
# Dynamic discovery mapping array variables on FreeBSD /bin/sh
index=1
chars="123456789abcdefghijklmnopqrstuvwxyz"
# Gather, size-calculate, sort, and loop GGUF models using FreeBSD native POSIX utilities
MATCHED_MODELS=$(find "$MODELPATH" -maxdepth 1 -name "*.gguf" 2>/dev/null | while read -r path; do
bytes=$(stat -f %z "$path")
name=$(basename "$path")
echo "$bytes $name"
done | sort -n)
old_ifs=$IFS
IFS='
'
for line in $MATCHED_MODELS; do
IFS=$old_ifs
bytes=$(echo "$line" | awk '{print $1}')
name=$(echo "$line" | cut -d' ' -f2-)
size_gb=$(awk -v b="$bytes" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
selector=$(echo "$chars" | cut -c "$index")
if [ -z "$selector" ]; then
break
fi
eval "MENU_KEY_$selector=\"$name\""
eval "MENU_SIZE_$selector=\"$size_gb\""
printf "%s) %6sG %s\n" "$selector" "$size_gb" "$name"
index=$((index + 1))
IFS='
'
done
IFS=$old_ifs
echo "0) Exit "
printf "Your choice: "
read choice
if [ "$choice" = "0" ] || [ -z "$choice" ]; then
echo "Exiting."
exit 0
fi
eval "SELECTED_MODEL=\$MENU_KEY_$choice"
eval "SELECTED_SIZE=\$MENU_SIZE_$choice"
if [ -z "$SELECTED_MODEL" ]; then
echo "❌ Invalid choice."
exit 1
fi
run_llama "$SELECTED_MODEL" "$SELECTED_SIZE"
Script to start aider:
sh:
#!/bin/sh
#export HF_HUB_DISABLE_XET=1
# Export env flags
export AIDER_MODEL="openai/kAI"
export AIDER_OPENAI_API_BASE="http://127.0.0.1:8080"
export OPENAI_API_BASE="http://127.0.0.1:8080"
export OPENAI_API_KEY="local-no-key-needed"
export AIDER_SYSTEM_PROMPT="
CRITICAL REQUIREMENT: Always write all of your C code, scripts,
configurations, and experiments into a single file named exactly
"new.c". Never invent other filenames.
You are an expert software developer specializing in FreeBSD.
Always ensure configurations, scripts, and code adhere strictly
to modern FreeBSD best practices and style guides (ZFS, Jails,
bhyve, pkg, and ports). Do not mix Linux commands or paths into
FreeBSD code. Do not mix code for OpenBSD into FreeBSD code.
Always assume the target environment compiles code exclusively with Clang/LLVM.
Ensure all configurations, scripts, and code adhere strictly to modern
FreeBSD best practices. Do not mix Linux commands or GCC-specific flags into code."
#export AIDER_NO_REPO_MAP="true"
#export TREE_SITTER_LANGUAGE_PACK_CACHE="/tmp/empty-syntax-cache"
# Check that AI-model is there
if ! nc -z -w 2 127.0.0.1 8080 > /dev/null 2>&1; then
echo "❌ Error: llama.cpp is not running on port 8080!"
exit 1
fi
# Activate the venv
. "$HOME/venvs/aider-env/bin/activate"
export PYTHONPATH="$HOME/aider"
rm -f .env
# Startup aider with the correct map disable flag
"$HOME/venvs/aider-env/bin/aider" \
--lint-cmd "c: clang -fsyntax-only" \
--test-cmd "$HOME/scripts/test-runner.sh" \
--map-tokens 0 \
--edit-format diff \
--no-show-model-warnings \
--yes-always
The test-runner.sh script that will run when /test is run in aider:
sh:
#!/bin/sh
# test_runner.sh
# 1. Automatically grab ALL .c files present in the current directory
ALL_FILES=$(ls *.c 2>/dev/null)
if [ -n "$ALL_FILES" ]; then
# 2. Pick a name for the binary based on the most recently modified C file
MAIN_FILE=$(git diff-tree --no-commit-id --name-only -r HEAD | grep "\.c$" | head -n 1)
if [ -z "$MAIN_FILE" ]; then
MAIN_FILE=$(ls -t *.c | head -n 1)
fi
BINARY="${MAIN_FILE%.c}"
echo "⚙ Automatically compiling all files ($ALL_FILES) into binary [$BINARY] using clang..."
# 3. Compile every single C file together
clang $ALL_FILES -o "$BINARY" && ./"$BINARY"
else
echo "❌ Error: Aider test runner couldn't locate any active C files."
exit 1
fi
You can run Ollama with Cuda on Freebsd using a Rocky Linux Podman container
The Nvidia Driver and Linuxulator libraries are mounted from Freebsd into the Podman container,
and you can start and stop Ollama with a Makefile by running make to start Ollama and make clean to stop it
Hey NapoleonWils0n - I have had your threads about podman projects on my screen for a week now.
I just need to find a little time and then I will look at the cuda uvm hack and see if I can puzzle it together with llama-cpp. If I understand it correctly llama-cpp is the engine in Ollama but I'm new to all this ai kerfuffling.
I think I saw that you have a thread about finding an x11 guinea pig and I will reply to that since my build-from-ports nvidia drivers forces me to use x11 on this old unbalanced hardware.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
www.wheresyoured.at
