Technical aspects of running local LLMs on FreeBSD

There is no FreeBSD-specific CUDA code, obviously.

we can agree on that
makes a change

As you said ports have to be patched

And there are some ports that havent been patched like Handbrake or Blender,
because you only have so much time, and as you said about Handbrake

FYI, there is no NVENC with Handbrake, because I didn't bother to submit the corresponding patches to the port.

That's correct isnt it.

So the project is addressing 2 issues

1) Ports that havent been patched to work with Cuda,
and installing the Linux version with Podman to get Cuda working.

2) Linux applications that require Cuda but cant be ported to Freebsd like Davinci Resolve because of licensing issues.

So if a port hasnt been patched the user has an alternative,
and can install the Linux version of the application with Podman and get it working with Cuda.

Rather than asking you or other devs to patch lots of different ports,
which you might not have the time for.

Surely thats a good thing from your point of view,
as it means less work patching.

Having pre configured applications that are easy to deploy,
and making things easier for user and giving them more choice is a good thing.

The end goal is getting an application working with Cuda for users,
we just have different ways of going about it.
 
ThisIsAGoodThread.png
 
To run a application/port that has been patched you have to prefix the command with nv-sglrun

for example to use ffmpeg with nvenc

Code:
nv-sglrun ffmpeg

script example

Code:
#!/bin/sh

#===============================================================================
# convert video to h265/aac
#===============================================================================


#===============================================================================
# script usage
#===============================================================================

usage()
{
# if argument passed to function echo it
[ -z "${1}" ] || echo "! ${1}"
# display help
echo "\
# convert video to h265/aac

$(basename "$0") -i input.mov -o output.mp4
-i infile.mov
-o outfile.mov :optional agument # if option not provided defaults to input-name.mp4"
exit 2
}


#===============================================================================
# error messages
#===============================================================================

NOTFILE_ERR='not a file'
INVALID_OPT_ERR='Invalid option:'
REQ_ARG_ERR='requires an argument'
WRONG_ARGS_ERR='wrong number of arguments passed to script'


#===============================================================================
# check number of aruments passed to script
#===============================================================================

[ $# -gt 0 ] || usage "${WRONG_ARGS_ERR}"


#===============================================================================
# getopts check options passed to script
#===============================================================================

while getopts ':i:o:h' opt
do
  case ${opt} in
     i) input="${OPTARG}"
    [ -f "${input}" ] || usage "${input} ${NOTFILE_ERR}";;
     o) output="${OPTARG}";;
     h) usage;;
     \?) usage "${INVALID_OPT_ERR} ${OPTARG}" 1>&2;;
     :) usage "${INVALID_OPT_ERR} ${OPTARG} ${REQ_ARG_ERR}" 1>&2;;
  esac
done
shift $((OPTIND-1))


#===============================================================================
# variables
#===============================================================================

input_nopath="${input##*/}"
input_name="${input_nopath%.*}"

# defaults for variables if not defined
output_default="${input_name}.mp4"


#===============================================================================
# functions
#===============================================================================

# h265 function
h265 () {
    nv-sglrun \
    ffmpeg \
    -hide_banner \
    -stats -v panic \
    -i "${input}" \
    -c:v hevc_nvenc \
    -pix_fmt p010le \
    -preset slow \
    -tier high \
    -rc vbr \
    -cq 22 \
    -b:v 0 \
    -maxrate 50M \
    -c:a aac \
    -b:a 320k \
    -ar 48000 \
    "${output:=${output_default}}"
}

# run the h265 function
h265 "${input}"

for gui applications you need to modify the desktop entry

for example to get obs studio working with nvenc

Code:
[i] Yes Master ? ls -l ~/.local/share/applications/com.obsproject.Studio.desktop
-rw-r--r--  1 djwilcox djwilcox 395  6 Jul 15:04 /home/djwilcox/.local/share/applications/com.obsproject.Studio.desktop

com.obsproject.Studio.desktop

Code:
[Desktop Entry]
Version=1.0
Name=OBS
GenericName=Streaming/Recording Software
Comment=Free and Open Source Streaming/Recording Software
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'
Icon=com.obsproject.Studio
Terminal=false
Type=Application
Categories=AudioVideo;Recorder;
StartupNotify=true
StartupWMClass=obs

notice the exec line

Code:
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'

as opposed to just running obs

Code:
Exec=obs --websocket_ipv4_only

However as i said if you try and install something like whisperx
which doesnt have a native Freebsd package with pip or conda with the Linuxulator or a Jail

Then it will fail to install because it will detect its running on Freebsd
and try and download Freebsd python wheels that dont exist

there are Freebsd python packages for torch

Code:
[i] Yes Master ? pkg search torch
py312-facenet-pytorch-2.5.3_4  Pretrained PyTorch face detection and recognition models
py312-lion-pytorch-0.2.4       PyTorch: Lion optimizer
py312-pytorch-2.12.1           PyTorch: Tensors and dynamic neural networks in Python
py312-pytorch-lightning-2.6.5  Lightweight PyTorch wrapper for ML researchers
py312-pytorchvideo-0.1.5_4     Video understanding deep learning library
py312-torch-geometric-2.8.0    Graph neural network library for PyTorch
py312-torchao-0.17.0           PyTorch: Package for applying ao techniques to GPU models
py312-torchaudio-2.11.0        PyTorch-based audio signal processing and machine learning library
py312-torchcodec-0.13.0        PyTorch media decoding and encoding
py312-torchdata-0.11.0         PyTorch: Composable data loading modules for PyTorch
py312-torchmetrics-1.9.0       PyTorch native metrics
py312-torchsde-0.2.6_2         SDE solvers and stochastic adjoint sensitivity analysis in PyTorch
py312-torchsummary-1.5.1_2     PyTorch: Model summary in PyTorch
py312-torchvision-0.27.0       PyTorch: Datasets, transforms and models specific to computer vision
pytorch-2.12.1                 Tensors and dynamic neural networks in Python (C++ library)
[

But they may not be the correct version for python application you are trying to install
and the application may require additional python libraries that dont have a Freebsd package

So the advantage of using Podman is you can set the container platform and os to Linux
which presents a Linux environment to applications like python so they download the python linux wheels

So for native Freebsd packages that have been patched to support nvenc for example
you can prefix the command with nv-sglrun or modify the desktop entry
 
BSD Jedi has some Ollama tutorial and a new one with a plug for some budget option cloud models.

I run 16GB vRAM nvidia on Debian Linux Ollama - rarely used, mainly as brain for Home Assistant....for fun.
I have good experience with Mixture of Experts models - gemma4. They really seem to use just parts of the model they actually need for given task.

Alas at work I have some good Ryzen AMD CPU, no vRAM to speak of but 64GB of DDR5 RAM (bought a year ago, yey) and it is surprisingly capable of running much bigger models than my home machine. Slow but who cares when crunching config files and not having conversations.

I also bought Google Coral TPUs on cheap but there is no chance I can make them work on naked FreeBSD - python libraries expect Linux. I was thinking of making a robot car with visual recognition on FreeBSD but all the python libraries are just made for Linux Rapsberry "drivers" too.

For more serious and private work 128GB is a must. But as OP pointed out if privacy is not a must, cloud models are better investment. Or rather 4,000 USD machine which is way dumber is just really bad investment right now.
 
Some quick and crude steps to get llama-cpp (version: 0.4.0-dev (build 10931, commit 3057bb66c) running with CUDA in the Linuxlator on FreeBSD 15.1.


llama-cpp with CUDA on the Linuxlator

# ----------------------------------------------------------------
# The FreeBSD side
# ----------------------------------------------------------------
# The necessary filesystems
# Note: Last line only necessary if you have models already
# on the BSD side. Change it to your path.

/etc/fstab
proc /proc procfs rw 0 0
linprocfs /compat/linux/proc linprocfs rw 0 0
linsysfs /compat/linux/sys linsysfs rw 0 0
tmpfs /compat/linux/dev/shm tmpfs rw,late,mode=1777 0 0
/home/<insertyourusernamehere>/AI-models /compat/linux/home/<insertyourusernamehere>/AI-models nullfs rw,late 0 0

# /etc/rc.conf
sudo sysrc linux_enable="YES"
sudo sysrc kld_list+="nvidia-modeset nvidia-drm linux64"

# Note: these are the drivers tested
nvidia-driver-595.99.02 NVIDIA graphics driver userland
nvidia-drm-612-kmod-595.99.02.1501000 NVIDIA DRM Kernel Module
nvidia-drm-kmod-595.99.02 NVIDIA DRM kernel module
nvidia-kmod-595.99.02.1501000 NVIDIA graphics driver kernel module

# Setting up the Linuxlator
sudo mkdir -p /compat/linux
sudo debootstrap --arch=amd64 jammy /compat/linux http://archive.ubuntu.com/ubuntu
sudo mount /compat/linux/proc
sudo mount /compat/linux/sys
sudo mount /compat/linux/dev/shm
sudo service linux start

# Below is version 595.99.02
cd /usr/ports/x11/linux-nvidia-libs
sudo make install
sudo mkdir -p /compat/linux/home/<insertyourusernamehere>/AI-models
sudo mount -t nullfs /home/<insertyourusernamehere>/AI-models /compat/linux/home/<insertyourusernamehere>/AI-models

# ----------------------------------------------------------------
# The Linux side
# ----------------------------------------------------------------
# Initializing Ubuntu
sudo chroot /compat/linux /bin/bash
apt update
apt-get install -y software-properties-common && add-apt-repository universe && apt update
apt install -y python3-pip python3-dev wget curl build-essential
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX_3.4.30
apt install -y ninja-build cmake
mkdir -p /opt/llama/bin

# The FreeBSD uvm modification
mkdir /tmp/freebsd-cuda
cd /tmp/freebsd-cuda
git clone https://github.com/NapoleonWils0n/freebsd-cuda.git .
cd /opt/llama/bin
cp /tmp/freebsd-cuda/base-build/uvm_ioctl_override/uvm_ioctl_override.c .
gcc -shared -fPIC -o /opt/llama/bin/dummy-uvm.so /opt/llama/bin/uvm_ioctl_override.c

# Installing cuda stuff
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
# These three below might come with cuda-toolkit as well
apt install -y cuda-cudart-12-8
apt install -y libcublas-12-8
apt install -y libnccl2

# Builing llama-cpp
cd /opt
git clone https://github.com/ggerganov/llama.cpp.git llama-build
cd llama-build
apt install -y cuda-compiler-12-8 cuda-toolkit-12-8
export PATH="/usr/local/cuda-12.8/bin:$PATH"
echo 'export PATH="/usr/local/cuda-12.8/bin:$PATH"' >> /root/.bashrc
export CUDAToolkit_ROOT="/usr/local/cuda-12.8"
echo 'export CUDAToolkit_ROOT="/usr/local/cuda-12.8"' >> /root/.bashrc
# Fix a 32/64 bit library issue
rm -f /usr/lib/libcuda.so
ln -s /usr/local/cuda-12.8/targets/x86_64-linux/lib/stubs/libcuda.so /usr/lib/libcuda.so
cd /opt/llama-build
rm -rf build
cmake -B build -G Ninja -DGGML_CUDA=ON
cmake --build build --config Release --target llama-server


# Running llama-cpp with cuda on FreeBSD
env LD_LIBRARY_PATH="/usr/local/cuda-12.8/lib64:/usr/lib64:/usr/lib/x86_64-linux-gnu" \
LD_PRELOAD="/opt/llama/bin/dummy-uvm.so" \
/opt/llama-build/build/bin/llama-server \
-m /home/<insertyourusernamehere>/AI-models/qwen2.5-coder-14b-instruct-q4_k_m.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 99


Note: Running nvidia-smi on the Linux side will not show CUDA - this is most likely because nvidia-smi not is built with the uvm modification.

Here's are some snippets of the output when llama-cpp start up:


0.00.014.956 I cmn common_param: common_params_print_info: build 10931 (3057bb66c) with GNU 11.2.0 for Linux x86_64
0.00.014.961 I cmn common_param: common_params_print_info: verbosity = 6 (adjust with the `-lv N` CLI arg)
0.00.014.961 I cmn common_param: device_info:
0.00.152.558 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3060 (12157 MiB, 11603 MiB free)
0.00.152.568 I cmn common_param: - CPU : Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz (16297 MiB, 16297 MiB free)
0.00.152.688 I cmn common_param: system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CUDA : ARCHS = 500,610,700,750,800,860,890,900,1200 | USE_GRAPHS = 1 | FA_QUANTS = q4_0-q4_0,q8_0-q8_0,f16-f16,bf16-bf16 | CPU : SSE3 = 1 | SSSE3 =
1 | AVX = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.152.695 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.152.825 I srv init: using 8 threads for HTTP server

....

0.00.264.148 I print_info: file size = 8.37 GiB (4.87 BPW)
0.00.264.669 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3060) (0000:01:00.0) - 11603 MiB free
0.00.357.956 D init_tokenizer: initializing tokenizer for type 2
0.00.392.019 I load: 0 unused tokens

....

0.00.515.627 I print_info: max token length = 256
0.00.515.872 I load_tensors: loading model tensors, this can take a while... (load_mode = none)
0.00.516.326 D load_tensors: layer 0 assigned to device CUDA0, is_swa = 0
0.00.516.327 D load_tensors: layer 1 assigned to device CUDA0, is_swa = 0
0.00.516.328 D load_tensors: layer 2 assigned to device CUDA0, is_swa = 0
0.00.516.328 D load_tensors: layer 3 assigned to device CUDA0, is_swa = 0


/grandpa
 
Some more technical aspects.

Running llama-cpp with CUDA on the Linuxlator and aider on the FreeBSD side with the same prompt (see #10) . Still on crap hardware.

The random models and how they did.

#SizeNameResultCUDAVULKAN
1)4.36Gqwen2.5-coder-7b-instruct-q4_k_m.ggufsuccess for load average but fail for temperature~ 62 t/s~ 54 t/s
2)
3)6.87Ggemma4-coding-Q4_K_M.ggufsuccess~ 33 t/s~ 30 t/s
4)
5)8.01GCodestral-22B-v0.1-IQ3_XXS.ggufsuccess~ 24 t/s~ 18 t/s
6)8.37Gqwen2.5-coder-14b-instruct-q4_k_m.ggufsuccess for temperature fail for load averages~ 32 t/s~ 30 t/s
7)
8)8.65GDevstral-Small-2-24B-Instruct-2512-UD-Q2_K_XL.ggufsuccess for temperature fails for load average~ 21 t/s~19 t/s
9)8.74GNorth-Mini-Code-1.0-UD-IQ1_M.ggufsuccess~ 72 t/s~ 54 t/s
a)
b)9.11Ggemma4-coding-Q6_K.ggufsuccess~ 27 t/s~25 t/s
c)9.11GQwen3.5-9B-Q8_0.gguffails stuck in a loop~ 34 t/s~25 t/s
d)14.84GQwen2.5-Coder-32B-Instruct-Q3_K_M.ggufsuccess for temperature fails slightly for load averages (2/3)~ 1.22 t/s~ 1 t/s

Notes:
-
Results may vary. Maybe it's the tweaking of the parameters. Especially suspect is the context window size.
- The feeling from #10 that Vulkan not always is giving memory back is totally gone. Switching between models does not cause out of device memory.
- In general it feels more smooth.

/grandpa
 
Edit: all scripts have been (hopefully) simplified Sep 17. This goes for #32 and #33.

Again with some technical aspects of local LLMs on FreeBSD. Sorry for scattered posts but I'm learning as I do this and there's alot of tweaking involved.

First a disclaimer.
- This post became so large I had to split it into 2 parts. see next post for the final script and some pictures.
- The scripts below are just a snapshot of the moment in time when this is being written. The evolution of llama-cpp and aider is going fast so expect change.
- The scripts uses sudo and llama-cpp runs as root (chroot) in the Linuxlator.
- The Scripts uses experimental flags like "tools all" for llama-cpp - be warned

This is the software from this snapshot in time:

  • aider 0.86.3.dev53+g5dc9490bb.d20260902 (pip installed in a venv and a modified requirements.txt)
  • FreeBSD 15.1-RELEASE-p3 releng/15.1-n283611-88e7371d9dc2 GENERIC amd64 including python3.12
  • linuxlator installed as described in #30
  • llama-cpp version: 0.4.0-dev (build 10931, commit 3057bb66c) installed as in #30
  • some AI-models from huggingface
What will these scripts do - briefly?
  • check computer RAM and NVRAM
  • read a list of models and their size from a specified directory
  • analyze those models for trained context window and layers
  • calculate optimal values for llama-cpp to fit inside the hardware constraints
  • show a list of models sorted with "fastest model first" and let user pick one to run
  • start aider with parameters coherent with llama-cpp
  • start claude with parameters coherent with llama-cpp
How do I run these scripts?

  1. Put the scripts in your shell path and make sure they are executable. Remember that user who runs needs to have sudo.
  2. run git-analyzer.sh
Note: A file name .git-analyzer.conf will be created and saved in user $HOME directory.

The scripts:

sh:
#!/bin/sh

# ==============================================================================
# 🌐 AI MASTER DASHBOARD
# ==============================================================================
CONFIG_FILE="$HOME/.git-analyzer.conf"

# Check if configuration profile is missing – if so, launch first-time setup!
if [ ! -f "$CONFIG_FILE" ]; then
    echo "⚠️  No configuration profile detected. Launching first-time setup..."
    /home/raggen/scripts/config-paths.sh
    if [ $? -ne 0 ]; then exit 1; fi
fi

# Load all saved environment variables directly into the shell
. "$CONFIG_FILE"

# Map to the internal variables expected by the rest of the script
SCRIPTS_DIR="$CONFIG_SCRIPTS_PATH"
MODELS_DIR="$CONFIG_MODELS_PATH"
WORKSPACE_DIR="$CONFIG_WORKSPACE_PATH"
AIDER_BIN_PATH="$CONFIG_AIDER_PATH"
CLAUDE_BIN_PATH="$CONFIG_CLAUDE_PATH"

SCRIPT_LAUNCHER="$SCRIPTS_DIR/ai-launcher.sh"

while true; do
echo "──────────────────────────────────────────────────────"
echo "🌐 AI Environment Master Dashboard"
echo "──────────────────────────────────────────────────────"
echo "💡 Active Workspace: $WORKSPACE_DIR"
echo "💡 Active AI Models: $MODELS_DIR"
echo "──────────────────────────────────────────────────────"
echo "Select execution path:"
echo "1) Run AI instantly using dynamic matrix optimiser"
echo "2) Change Configuration Paths"
echo "0) Exit"
echo "──────────────────────────────────────────────────────"
printf "Your choice: "
read -r menu_choice

case "$menu_choice" in

    1)
        echo "🚀 Invoking matrix pipeline..."
        cd "$WORKSPACE_DIR" || exit 1
        # Removed the DEFAULT_KB parameter. Arguments shifted left by 1.
        sh "$SCRIPT_LAUNCHER" "$SCRIPTS_DIR" "$MODELS_DIR" "$AIDER_BIN_PATH" "$CLAUDE_BIN_PATH"
        ;;

    2)
        # Invoke the configuration script live
        /home/raggen/scripts/config-paths.sh
      
        # Immediately reload the new paths into memory to update the dashboard live
        if [ -f "$CONFIG_FILE" ]; then
            . "$CONFIG_FILE"
            SCRIPTS_DIR="$CONFIG_SCRIPTS_PATH"
            MODELS_DIR="$CONFIG_MODELS_PATH"
            WORKSPACE_DIR="$CONFIG_WORKSPACE_PATH"
            AIDER_BIN_PATH="$CONFIG_AIDER_PATH"
            CLAUDE_BIN_PATH="$CONFIG_CLAUDE_PATH"
            SCRIPT_LAUNCHER="$SCRIPTS_DIR/ai-launcher.sh"
        fi
        ;;

    0|*)
        echo "Exiting Dashboard. Goodbye."
        exit 0
        ;;
esac

done
sh:
#!/bin/sh

echo "──────────────────────────────────────────────────────"
echo "🖥️  FreeBSD 15 Hardware Architecture Discovery"
echo "──────────────────────────────────────────────────────"

# 1. DISCOVER SYSTEM RAM
PHYSMEM_BYTES=$(sysctl -n hw.physmem 2>/dev/null)

if [ -n "$PHYSMEM_BYTES" ]; then
    TOTAL_RAM_GB=$(awk -v b="$PHYSMEM_BYTES" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
    echo "🧠 System RAM Status:"
    echo "   • Total Physical RAM : $TOTAL_RAM_GB GB"
else
    echo "❌ Error: Unable to query physical system memory via sysctl."
    TOTAL_RAM_GB="0.00"
fi

echo "──────────────────────────────────────────────────────"

# 2. DISCOVER NVIDIA GRAPHICS VRAM
echo "🎮 NVIDIA Graphics VRAM Status:"

if [ ! -c /dev/nvidia0 ]; then
    echo "   ❌ Error: NVIDIA hardware device node (/dev/nvidia0) is missing!"
    TOTAL_VRAM_GB="0.00"
else
    NV_DRIVER_VERSION=$(sysctl -n hw.nvidia.version 2>/dev/null)
    if [ -n "$NV_DRIVER_VERSION" ]; then
        echo "   • Kernel Driver      : NVIDIA UNIX x86_64 ($NV_DRIVER_VERSION)"
    fi

    # Read the card model description string directly from the sysctl subsystem
    GPU_MODEL=$(sysctl -n hw.nvidia.0.description 2>/dev/null)
    if [ -n "$GPU_MODEL" ]; then
        echo "   • Hardware Model     : $GPU_MODEL"
    fi

    # MODERN FREEBSD 15 FALLBACK: Query pciconf for the active NVIDIA card memory map bounds
    # This reads the actual BAR memory windows allocated to the card on the PCIe bus
    VRAM_BYTES=$(pciconf -lv | grep -A 4 "vgapci" | grep -i "nvidia" -A 4 2>/dev/null | grep "bar.*vram" | sed -E 's/.*vram[[:space:]]*([0-9]+).*/\1/')
  
    if [ -z "$VRAM_BYTES" ]; then
        # Secondary fallback: Extract memory footprint sizing directly from the hardware description line
        VRAM_MB=$(sysctl -a | grep "nvidia" | grep -i "vram" | head -n 1 | awk '{print $NF}' | sed 's/[^0-9]//g')
    else
        VRAM_MB=$((VRAM_BYTES / 1024 / 1024))
    fi

    # If sysctl and pciconf both return empty, match against your known 12GB physical configuration
    if [ -z "$VRAM_MB" ] || [ "$VRAM_MB" -eq 0 ]; then
        VRAM_MB=12288
    fi

    TOTAL_VRAM_GB=$(awk -v mb="$VRAM_MB" 'BEGIN {printf "%.2f", mb / 1024}')
    echo "   • Dedicated VRAM     : $TOTAL_VRAM_GB GB ($VRAM_MB MB)"
fi

echo "──────────────────────────────────────────────────────"
echo "✅ Discovery phase complete."
echo "──────────────────────────────────────────────────────"

export INVENTORIED_RAM="$TOTAL_RAM_GB"
export INVENTORIED_VRAM="$TOTAL_VRAM_GB"
#!/usr/local/bin/python3.12
import sys
import struct

def read_gguf_metadata(filepath):
ctx_val, blocks_val = None, None
try:
with open(filepath, "rb") as f:
# 1. Verify Magic Header "GGUF"
magic = f.read(4)
if magic != b"GGUF":
return None, None

# 2. Read Version (uint32) - Extracts the integer out of the unpack tuple
version = struct.unpack("<I", f.read(4))[0]
if version not in set((2, 3)):
return None, None

# 3. Read metadata counts safely using version boundaries
tensor_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
kv_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]

# 4. Iterate over key-value metadata blocks
for _ in range(kv_count):
key_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
key = f.read(key_len).decode("utf-8", errors="ignore")
val_type = struct.unpack("<I", f.read(4))[0]

# Helper closure to skip or evaluate values based on GGUF type IDs
def skip_value(t):
# 0=uint8, 1=int8, 7=bool
if t in set((0, 1, 7)):
f.seek(1, 1)
# 2=uint16, 3=int16
elif t in set((2, 3)):
f.seek(2, 1)
# 4=uint32, 5=int32, 6=float32
elif t in set((4, 5, 6)):
val = struct.unpack("<I", f.read(4))[0]
return val
# 8=string
elif t == 8:
s_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
f.seek(s_len, 1)
# 9=array
elif t == 9:
arr_type = struct.unpack("<I", f.read(4))[0]
arr_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
for _ in range(arr_len):
skip_value(arr_type)
# 10=uint64, 11=int64, 12=float64, 13=uint64, 14=int64
elif t in set((10, 11, 12, 13, 14)):
f.seek(8, 1)
return None

res = skip_value(val_type)

# 5. Capture context size and block layers out of matching keys
if "context_length" in key and res is not None:
ctx_val = res
elif "block_count" in key and res is not None:
blocks_val = res

# Break loop immediately if both core targets are gathered
if ctx_val is not None and blocks_val is not None:
break

except Exception:
pass

return ctx_val, blocks_val

if __name__ == "__main__":
if len(sys.argv) > 1:
# Pass index 1 to isolate the path argument string
ctx, blocks = read_gguf_metadata(sys.argv[1])
ctx_str = str(ctx) if ctx else "Unknown"
blocks_str = str(blocks) if blocks else "Unknown"
print(f"{ctx_str} {blocks_str}")

Python:
sh:
#!/bin/sh

# ==============================================================================
# ⚙️ AI CONFIGURATION LAYER ENVIRONMENT CONFIGURATOR 
# ==============================================================================
CONFIG_FILE="$HOME/.git-analyzer.conf"

echo "──────────────────────────────────────────────────────"
echo "⚙️ AI Cluster Configuration Setup"
echo "──────────────────────────────────────────────────────"

# Load existing environment variables if the configuration file already exists
if [ -f "$CONFIG_FILE" ]; then
    . "$CONFIG_FILE"
fi

# 1. SCRIPTS PATH
printf "Enter full path to scripts directory [%s]: " "${CONFIG_SCRIPTS_PATH:-/home/user/scripts}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    # Strip any accidental trailing forward slashes
    CONFIG_SCRIPTS_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_SCRIPTS_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_SCRIPTS_PATH' does not exist."
    exit 1
fi

# 2. MODELS PATH
printf "Enter full path to GGUF models directory [%s]: " "${CONFIG_MODELS_PATH:-/home/user/AI-models}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    # Strip any accidental trailing forward slashes
    CONFIG_MODELS_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_MODELS_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_MODELS_PATH' does not exist."
    exit 1
fi

# 3. WORKSPACE PATH
printf "Enter full path to local development workspace [%s]: " "${CONFIG_WORKSPACE_PATH:-/home/user/Projects}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    # Strip any accidental trailing forward slashes
    CONFIG_WORKSPACE_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_WORKSPACE_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_WORKSPACE_PATH' does not exist."
    exit 1
fi

# 4. AIDER BINARY PATH
printf "Enter full path to Aider executable [%s]: " "${CONFIG_AIDER_PATH:-/home/user/venvs/aider-env/bin/aider}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_AIDER_PATH="$USER_INPUT"
fi
if [ ! -f "$CONFIG_AIDER_PATH" ]; then
    echo "❌ Error: Executable '$CONFIG_AIDER_PATH' does not exist."
    exit 1
fi

# 5. CLAUDE BINARY PATH
printf "Enter full path to Claude Code executable [%s]: " "${CONFIG_CLAUDE_PATH:-/home/user/.local/bin/claude}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_CLAUDE_PATH="$USER_INPUT"
fi
if [ ! -f "$CONFIG_CLAUDE_PATH" ]; then
    echo "❌ Error: Executable '$CONFIG_CLAUDE_PATH' does not exist."
    exit 1
fi

# Write compiled environment routing variables out to the hidden profile file
cat << EOC > "$CONFIG_FILE"
CONFIG_SCRIPTS_PATH="$CONFIG_SCRIPTS_PATH"
CONFIG_MODELS_PATH="$CONFIG_MODELS_PATH"
CONFIG_WORKSPACE_PATH="$CONFIG_WORKSPACE_PATH"
CONFIG_AIDER_PATH="$CONFIG_AIDER_PATH"
CONFIG_CLAUDE_PATH="$CONFIG_CLAUDE_PATH"
EOC

echo "──────────────────────────────────────────────────────"
echo "✅ Configuration layer compiled and saved to ~/.git-analyzer.conf"
echo "──────────────────────────────────────────────────────"
sh:
#!/bin/sh

# ==============================================================================
# 🧮 AI MASTER MATRIX & CONTEXT CALCULATOR
# ==============================================================================
# Purpose: Inherits automated directory pointers from the launcher pipeline
#          and calculates the speed scoring matrix lines by dynamically
#          balancing maximal VRAM utilisation against high-speed execution.
#          Appends the native trained context for UI visibility.
# ==============================================================================

# Positional arguments inherited cleanly from the launcher script
SCRIPTS_DIR="$1"
MODEL_DIR="$2"

if [ -z "$SCRIPTS_DIR" ] || [ -z "$MODEL_DIR" ]; then
    echo "❌ Calculation Error: Missing critical orchestration path arguments." >&2
    exit 1
fi

# Define secondary tracking nodes
SCRIPT_DISCOVERY="$SCRIPTS_DIR/hw-discovery.sh"
SCRIPT_READER="$SCRIPTS_DIR/gguf-reader.py"
PYTHON_BIN="python3.12"
SYSTEM_VRAM_OVERHEAD=1.80

if [ -f "$SCRIPT_DISCOVERY" ]; then
    . "$SCRIPT_DISCOVERY" >/dev/null 2>&1
else
    echo "❌ System Error: Hardware discovery module ($SCRIPT_DISCOVERY) not found!" >&2
    exit 1
fi

TOTAL_VRAM=${INVENTORIED_VRAM:-12.00}
AVAILABLE_VRAM=$(awk -v vram="$TOTAL_VRAM" -v ov="$SYSTEM_VRAM_OVERHEAD" 'BEGIN {printf "%.2f", vram - ov}')

MATCHED_MODELS=$(find "$MODEL_DIR" -maxdepth 1 -name "*.gguf" 2>/dev/null | sort)
if [ -z "$MATCHED_MODELS" ]; then
    echo "ℹ  No .gguf targets found inside $MODEL_DIR." >&2
    exit 0
fi

old_ifs=$IFS
IFS='
'
for model_path in $MATCHED_MODELS; do
    IFS=$old_ifs
    filename=$(basename "$model_path")
    bytes=$(stat -f %z "$model_path")
    size_gb=$(awk -v b="$bytes" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
  
    METADATA=$($PYTHON_BIN "$SCRIPT_READER" "$model_path" 2>/dev/null)
    trained_ctx=$(echo "$METADATA" | awk '{print $1}')
    layer_count=$(echo "$METADATA" | awk '{print $2}')
  
    # Fall back to safe structural boundaries if metadata extraction yields Unknown
    if [ -z "$trained_ctx" ] || [ "$trained_ctx" = "Unknown" ]; then trained_ctx=32768; fi
    if [ -z "$layer_count" ] || [ "$layer_count" = "Unknown" ]; then layer_count=40; fi

    # FIXED: Removed the empty target variable checking lines that triggered the shell 'bad number' error

    # Pass parameters to awk to execute balanced mathematical allocations
    awk -v av_vram="$AVAILABLE_VRAM" -v layers="$layer_count" -v m_size="$size_gb" -v max_trained="$trained_ctx" -v fname="$filename" '
    BEGIN {
        cache_type = "f16"; cache_factor = 0.0065;
        vram_left_for_cache = av_vram - m_size;
      
        # Scenario A: Model fits inside available VRAM (Prioritise full GPU acceleration)
        if (vram_left_for_cache >= 0) {
            # Compute the absolute mathematical ceiling for high-precision f16 context
            max_vram_ctx = int((vram_left_for_cache * 1000) / (layers * cache_factor));
            ctx = int(max_vram_ctx / 1024) * 1024;
          
            # If f16 cache pinches the context below 16k, drop to q4_0 to maximize token depth
            if (ctx < 16384) {
                cache_type = "q4_0"; cache_factor = 0.0017;
                max_vram_ctx = int((vram_left_for_cache * 1000) / (layers * cache_factor));
                ctx = int(max_vram_ctx / 1024) * 1024;
            }
          
            # Enforce a secure operational baseline floor of 8,192 tokens for usability
            if (ctx < 8192) ctx = 8192;
          
            # Clamp the context window strictly to the model native trained boundary
            if (ctx > max_trained) ctx = max_trained;

            ngl = layers + 5; mode = "mlock";
            speed_score = int(20000 - (m_size * 100));
        } else {
            # Scenario B: VRAM overflow fallback (mmap hybrid mode)
            # Allocate a balanced, static 16k context window for memory stability
            cache_type = "q4_0"; cache_factor = 0.0017;
            ctx = 16384;
            if (ctx > max_trained) ctx = max_trained;

            cache_vram_cost = (ctx / 1000) * layers * cache_factor;
            usable_vram_for_layers = av_vram - cache_vram_cost;
            if (usable_vram_for_layers < 0) usable_vram_for_layers = 0;
          
            vram_ratio = usable_vram_for_layers / m_size;
            ngl = int(layers * vram_ratio); if (ngl < 0) ngl = 0;
            mode = "mmap";
            speed_score = int(1000 * (ngl / layers));
        }
      
        # Hard system safety rail to prevent severe server faults
        if (ctx < 4096) ctx = 4096;
      
        gsub(/ /, "___SPACE___", fname);
        printf "%05d %s %d %d %s %s %s %s\n", speed_score, m_size, ngl, ctx, cache_type, mode, fname, max_trained;
    }'
    IFS='
'
done
IFS=$old_ifs

There is one more script but dues to size restrictions it will be in next post.

/grandpa
 
Last edited:
Edit: all scripts have been (hopefully) simplified Sep 17. This goes for #32 and #33.

Continued from previous post.

Oh, I almost forgot. I changed the llama-cpp to log to /var/logs. This is required for that:
sudo touch /var/log/llama-server.log
sudo chown user /var/log/llama-server.log


sh:
#!/bin/sh

# ==============================================================================
# 🚀 AI INTERACTIVE INTERFACE & DAEMON LAUNCHER
# ==============================================================================
# Purpose: Automatically inherits execution path variables passed directly
#          from the parent orchestrator dashboard. Zero hardcoded paths.
# ==============================================================================

# Catch dynamic paths propagated from git-analyzer via parameters
SCRIPTS_DIR="$1"
MODEL_ROOT_DIR="$2"
AIDER_BIN="$3"
CLAUDE_BIN="$4"

if [ -z "$SCRIPTS_DIR" ] || [ -z "$MODEL_ROOT_DIR" ] || [ -z "$AIDER_BIN" ]; then
    echo "❌ Interface Error: Missing necessary environment parameter inputs. Aborting."
    exit 1
fi

# Set unified dynamic local script path bindings
SCRIPT_CALCULATOR="$SCRIPTS_DIR/ctx-calculator.sh"
TEST_RUNNER="$SCRIPTS_DIR/test-runner.sh"

# Static Host System Environment Settings
SYSTEM_LOG_FILE="/var/log/llama-server.log"

# Static Container & Linuxulator Architectural Nodes
CHROOT_PATH="/compat/linux"
CHROOT_ENV="/usr/bin/env"
LLAMA_SERVER_BIN="/opt/llama-build/build/bin/llama-server"
DUMMY_UVM_HOOK="/opt/llama/bin/dummy-uvm.so"

# Network Parameters
SERVER_HOST="0.0.0.0"
SERVER_PORT="8080"
AIDER_TARGET_HOST="127.0.0.1"

# ==============================================================================
# 🆕 STEP 1: INTERACTIVE CLIENT INTERFACE SELECTION FORK
# ==============================================================================
echo "────────────────────────────────────────────────────────────────────────"
echo "Select your active AI coding interface:"
echo "1) Aider (Standard diff edit format toolchain)"
echo "2) Claude Code (Anthropic agent CLI toolchain)"
echo "────────────────────────────────────────────────────────────────────────"
printf "Your choice: "
read -r client_choice

if [ -z "$client_choice" ]; then
    client_choice="1"
fi

# Set dynamic tools variable based on selection to prevent MCP parsing conflicts
if [ "$client_choice" = "1" ]; then
    LLAMA_TOOLS="--tools all"
else
    LLAMA_TOOLS=""
fi

# FETCH RAW PROFILE DATA ENGINE AND EXECUTE SORTING PIPELINE
RAW_DATA=$("$SCRIPT_CALCULATOR" "$SCRIPTS_DIR" "$MODEL_ROOT_DIR")
if [ $? -ne 0 ] || [ -z "$RAW_DATA" ]; then
    echo "❌ Error: Matrix calculation pipeline failed."
    exit 1
fi

SORTED_MATRIX=$(echo "$RAW_DATA" | sort -k1,1r -n)

echo "────────────────────────────────────────────────────────────────────────────────"
printf "%-6s  %-8s  %-5s  %-6s  %-6s  %-5s  %-6s  %s\n" "INDEX" "VRAM/RAM" "NGL" "CTX" "CTXtr" "CACHE" "MODE" "OPTIMISED COMPATIBLE TARGET"
echo "────────────────────────────────────────────────────────────────────────────────"

# RENDER INTERACTIVE SPEED-SORTED DESIGN VIEW
index=1
chars="123456789abcdefghijklmnopqrstuvwxyz"
old_ifs=$IFS
IFS='
'
MATRIX_DATABASE=""

for row in $SORTED_MATRIX; do
    IFS=$old_ifs
    size_gb=$(echo "$row" | awk '{print $2}')
    ngl=$(echo "$row" | awk '{print $3}')
    ctx=$(echo "$row" | awk '{print $4}')
    cache=$(echo "$row" | awk '{print $5}')
    mode=$(echo "$row" | awk '{print $6}')
    raw_filename=$(echo "$row" | awk '{print $7}')
    ctxtr=$(echo "$row" | awk '{print $8}')
    filename=$(echo "$raw_filename" | sed 's/___SPACE___/ /g')

    selector=$(echo "$chars" | cut -c "$index")
    if [ -z "$selector" ]; then selector="-"; fi

    MATRIX_DATABASE="${MATRIX_DATABASE}${selector}|${filename}|${ctx}|${ngl}|${mode}|${cache}
"

    printf "%-6s  %-8s  %-5s  %-6s  %-6s  %-5s  %-6s  %s\n" "${selector})" "${size_gb}G" "$ngl" "$ctx" "$ctxtr" "$cache" "$mode" "$filename"
    index=$((index + 1))
    IFS='
'
done
IFS=$old_ifs

echo "0)      Exit"
echo "────────────────────────────────────────────────────────────────────────────────"
printf "Your choice: "
read -r choice

if [ "$choice" = "0" ] || [ -z "$choice" ]; then
    echo "Exiting."
    exit 0
fi

SELECTED_ROW=$(echo "$MATRIX_DATABASE" | grep "^${choice}|" | head -n 1)

if [ -z "$SELECTED_ROW" ]; then
    echo "❌ Input Error: Invalid matrix selection."
    exit 1
fi

SELECTED_MODEL=$(echo "$SELECTED_ROW" | cut -d'|' -f2)
SELECTED_CTX=$(echo "$SELECTED_ROW" | cut -d'|' -f3)
SELECTED_NGL=$(echo "$SELECTED_ROW" | cut -d'|' -f4)
SELECTED_MODE=$(echo "$SELECTED_ROW" | cut -d'|' -f5)
SELECTED_CACHE=$(echo "$SELECTED_ROW" | cut -d'|' -f6)

echo "────────────────────────────────────────────────────────────────────────"
echo "🚀 Launching C++ CUDA Server silently in the background (LAN Active)..."
echo "────────────────────────────────────────────────────────────────────────"

truncate -s 0 "$SYSTEM_LOG_FILE"

# INVOKE HEADLESS DAEMON ROUTED OVER MULTI-LAN PORTS
sudo chroot "$CHROOT_PATH" "$CHROOT_ENV" \
    PATH="/usr/local/cuda-12.8/bin:$PATH" \
    CUDAToolkit_ROOT="/usr/local/cuda-12.8" \
    LD_LIBRARY_PATH="/usr/local/cuda-12.8/lib64:/usr/lib64:/usr/lib/x86_64-linux-gnu" \
    LD_PRELOAD="$DUMMY_UVM_HOOK" \
    "$LLAMA_SERVER_BIN" \
        -t 3 -tb 4 -b 512 -ub 256 --parallel 1 --context-shift \
        -n 4096 --repeat-penalty 1.15 --repeat-last-n 64 --temp 0.3 -fa on \
        --chat-template chatml \
        --ctx-size "$SELECTED_CTX" \
        --n-gpu-layers "$SELECTED_NGL" \
        --load-mode "$SELECTED_MODE" \
        --cache-type-k "$SELECTED_CACHE" \
        --cache-type-v "$SELECTED_CACHE" \
        --host "$SERVER_HOST" --port "$SERVER_PORT" $LLAMA_TOOLS \
        -m "$MODEL_ROOT_DIR/$SELECTED_MODEL" > "$SYSTEM_LOG_FILE" 2>&1 &

SERVER_PID=$!

# HANDSHAKE NETWORK PORT MONITOR (Pure Sockstat Probing)
echo "⏳ Map-streaming layers into memory tracks (LAN Binding active)..."
ready=0
for i in $(seq 1 120); do
    if sockstat -4 -l -P tcp -p "$SERVER_PORT" | grep -q "$SERVER_PORT" 2>/dev/null; then
        sleep 5
        ready=1
        break
    fi
    printf "."
    sleep 0.5
done
printf "\n"

if [ $ready -eq 0 ]; then
    echo "❌ Error: Server initialization network handshake timed out."
    kill "$SERVER_PID" 2>/dev/null; exit 1
fi

# EXECUTE TARGET INTERFACE CLIENT
echo "✅ Server network handshake verified. Listening on port $SERVER_PORT."
echo "────────────────────────────────────────────────────────────────────────"

# Capture the native FreeBSD host wall-clock string live
CURRENT_TIME_STRING=$(date "+%A, %d %B %Y at %H:%M %Z")

# Generate the universal environment instruction file with strict response bounds
cat << EOF > CLAUDE.md
# FreeBSD AI Cluster Environment Instructions
- Host Operating System: FreeBSD 15 (CUDA Hardware Accelerated)
- Current Real-World Timestamp: $CURRENT_TIME_STRING

## Operational Rules (Strict)
- Answer the user's direct question accurately and concisely using the timestamp above.
- STOP generating text immediately after answering. Do not speculate, suggest next steps, or provide structural code unless explicitly asked to do so.
- Keep your output minimal and yield back control to the prompt.
EOF

case "$client_choice" in
    2)
        SCRIPT_CLAUDE="$SCRIPTS_DIR/run-claude.sh"
        if [ -f "$SCRIPT_CLAUDE" ]; then
            sh "$SCRIPT_CLAUDE" "$SELECTED_MODEL" "$SELECTED_CTX" "$CLAUDE_BIN"
        else
            echo "❌ System Error: Claude runner script ($SCRIPT_CLAUDE) is missing!"
            sudo kill "$SERVER_PID" 2>/dev/null
            exit 1
        fi
        ;;

    1|*)
        SCRIPT_AIDER="$SCRIPTS_DIR/run-aider.sh"
        if [ -f "$SCRIPT_AIDER" ]; then
            sh "$SCRIPT_AIDER" "$SELECTED_MODEL" "$SELECTED_CTX" "$AIDER_BIN"
        else
            echo "❌ System Error: Aider runner script ($SCRIPT_AIDER) is missing!"
            sudo kill "$SERVER_PID" 2>/dev/null
            exit 1
        fi
        ;;
esac

# ==============================================================================
# 🧳 INTERACTIVE TEARDOWN MANAGEMENT PIPELINE
# ==============================================================================
echo "────────────────────────────────────────────────────────────────────────"
echo "⚠️  AI Client Interface session closed."
echo "────────────────────────────────────────────────────────────────────────"
echo "Select exit action (Background llama-server will be terminated):"
echo "1) Stop server and return to Master Dashboard"
echo "2) Stop server and exit completely"
echo "────────────────────────────────────────────────────────────────────────"
printf "Your choice: "
read -r teardown_choice

# Shut down the background server unconditionally since it should never remain running
echo "Shutting down background server cluster..."
sudo kill "$SERVER_PID" 2>/dev/null

case "$teardown_choice" in
    2)
        echo "Exiting all environments. Goodbye."
        # Forces a hard exit bypassing the parent dashboard shell loop completely
        kill -9 $PPID
        exit 0
        ;;
    1|*)
        echo "Returning to Master Dashboard..."
        ;;
esac
sh:
#!/bin/sh

# ==============================================================================
# 🚀 FREEBSD NATIVE HARDWARE-ACCELERATED AIDER CLIENT RUNNER
# ==============================================================================
SELECTED_MODEL="$1"
SELECTED_CTX="$2"
AIDER_BIN_EXEC="$3"

if [ -z "$SELECTED_MODEL" ] || [ -z "$SELECTED_CTX" ] || [ -z "$AIDER_BIN_EXEC" ]; then
    echo "❌ Aider Pipeline Error: Missing critical orchestration parameters. Aborting."
    exit 1
fi

# Local Script Path Resolution Nodes
SCRIPTS_DIR="/home/raggen/scripts"
TEST_RUNNER="$SCRIPTS_DIR/test-runner.sh"
AIDER_TARGET_HOST="127.0.0.1"
SERVER_PORT="8080"

# 1. MAP CORE API ROUTING AND MODEL SELECTION LAYERS
export OPENAI_API_BASE="http://${AIDER_TARGET_HOST}:${SERVER_PORT}/v1"
export OPENAI_API_KEY="dummy"
export AIDER_MODEL="openai/$SELECTED_MODEL"

# 🔥 FORCE CONCISE RESPONSES AND STOP OVER-REASONING:
# Sets the model temperature to near-zero to guarantee strict,
# non-speculative, and direct answers without spinning into thinking loops.
export AIDER_TEMPERATURE="0.1"

# ==============================================================================
# 2. APPLY HARD OPTIMISATION AND MEMORY RESTRICTIONS
# ==============================================================================
export AIDER_MAX_CONTEXT_WINDOW="$SELECTED_CTX"
export AIDER_TEMPERATURE="0.1"
export AIDER_MAP_TOKENS="0"
export AIDER_MAX_CHAT_HISTORY_TOKENS="8192"

# ==============================================================================
# 5. EXECUTE THE NATIVE FREEBSD AIDER ENGINE FLOW
# ==============================================================================
echo "🤖 Initialising Aider Client Session"
echo "────────────────────────────────────────────────────────────────────────"

"$AIDER_BIN_EXEC" \
 --lint-cmd "c: clang -fsyntax-only" \
 --test-cmd "$TEST_RUNNER" \
 --git \
 --edit-format diff --no-show-model-warnings --yes-always


exit 0
sh:
#!/bin/sh

# ==============================================================================
# 🤖 CLAUDE CODE CLI RUNNER
# ==============================================================================
# Purpose: Dynamically maps Anthropic environmental layers to perfectly match
#          the hardware and token boundaries calculated by the matrix engine.
#          Utilises full parameter paths and aggressive context compression.
# ==============================================================================

# Inherit dynamic runtime metrics forwarded from the parent launcher pipeline
SELECTED_MODEL="$1"
SELECTED_CTX="$2"
CLAUDE_BIN_PATH="$3"

if [ -z "$SELECTED_MODEL" ] || [ -z "$SELECTED_CTX" ] || [ -z "$CLAUDE_BIN_PATH" ]; then
    echo "❌ Execution Error: Missing dynamic context or executable matrix parameters." >&2
    exit 1
fi

# 1. DYNAMICALLY COMPUTE CLAUDE BOUNDARIES BASED ON ENGINE CALCULATIONS
# We calculate the safe compaction threshold at exactly 80% of the active server context
# to provide an aggressive buffer against heavy initial Git repository index states.
COMPACT_WINDOW=$(awk -v ctx="$SELECTED_CTX" 'BEGIN {printf "%d", ctx * 0.80}')

# ==============================================================================
# 2. MAP THE ADAPTIVE ANTHROPIC AGENT ENVIRONMENT LAYERS
# ==============================================================================
export ANTHROPIC_MODEL="claude-sonnet-5"
export ANTHROPIC_BASE_URL="http://127.0.0.1:8080"
export ANTHROPIC_DEFAULT_SONNET_MODEL="claude-sonnet-5"
export ANTHROPIC_DEFAULT_OPUS_MODEL="claude-sonnet-5"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-sonnet-5"
export ANTHROPIC_AUTH_TOKEN="ollama"
export ANTHROPIC_SMALL_FAST_MODEL="claude-sonnet-5"

# Apply the dynamic context window settings directly to Claude Code's memory engine
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="$COMPACT_WINDOW"
export CLAUDE_AUTOCOMPACT_PCT_OVERRIDE="80"
export CLAUDE_CODE_DISABLE_1M_CONTEXT="1"
export CLAUDE_CODE_MAX_OUTPUT_TOKENS="4096"

# 🔥 THE TRUE LOCAL LOCAL-MODEL FIX:
# Strictly disable cloud-dependent fast-modes to ensure Claude Code
# parses local native JSON tool schemas instead of hallucinating.
export CLAUDE_CODE_DISABLE_FAST_MODE="1"
export CLAUDE_CODE_MAX_CHAT_HISTORY_TOKENS="8192"
export CLAUDE_CODE_DISABLE_GIT_INDEXING="1"

# Set Temperature to low precision to guarantee perfect syntax formatting during edits
export CLAUDE_CODE_MAX_RETRIES="5"
export DISABLE_PROMPT_CACHING="1"
export CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING="1"
export MAX_THINKING_TOKENS="0"
export DISABLE_INTERLEAVED_THINKING="1"
export CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY="1"
export DISABLE_TELEMETRY="1"
export CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY="1"
export ENABLE_TOOL_SEARCH="auto"
export CLAUDE_CODE_GIT_BASH_PATH="/usr/local/bin/bash"

# 🔥 FIXES THE CHANNEL SPINNING LOOP & SILENT TIMEOUTS:
export CLAUDE_CODE_DISABLE_CHANNELS="1"
export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS="1"
export MCP_PROTOCOL_NEGOTIATION="legacy"
export MCP_TOOL_TIMEOUT="5000"

echo ""
echo "--- Llama.cpp Configuration Loaded ---"
echo "Model Selected : $SELECTED_MODEL (Mapped to Sonnet-5 Profile)"
echo "LAN Endpoint   : $ANTHROPIC_BASE_URL"
echo "Calculated CTX : $SELECTED_CTX tokens"
echo "Compact Buffer : $COMPACT_WINDOW tokens (Triggers auto-rolling shift at 80%)"
echo "Executable     : $CLAUDE_BIN_PATH"
echo "────────────────────────────────────────────────────────────────────────"

# 3. FIXED: Fire up Claude Code natively utilizing the dynamic variable path parameter!
"$CLAUDE_BIN_PATH" --permission-mode acceptEdits



/grandpa
 
Last edited:
So, while the linuxlator is installed (see #30) it's quite nice to be able to run the latest version of claude.

This can be done by doing 3 steps providing you have node installed.
  1. sudo npm install -g --allow-scripts=@anthropic-ai/claude-code @anthropic-ai/claude-code
  2. sudo mkdir -p /lib/x86_64-linux-gnu
  3. sudo ln -sf /compat/linux/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
What this does is that node grabs the latest version of claude and installs it to user profile. That version of claude has a dependency to a linux library. Since we have the linuxlator we have that library. Create the folder and symlink to trick claude into running.


/grandpa
 
Last edited:
Great topic for a thread, thanks cracauer@.

Here's some technical aspects for running AI models locally on unbalanced crap hardware with llama-cpp, some random AI models from huggingface and aider.

The crap Hardware:

CPU Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz ~ 2011
16 Gb DD3 RAM @ 1600 ~ 2007
RTX 3060 with 12 Gb NVRAM ~ 2021

Installations:

llama-cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake -DGGML_VULKAN=1 ..
cmake --build . --config Release -j 3
cp llama* lib*.so* ~/.local/bin/


aider - this one is a bit trickier - git clone the source and then pip install in a venv and fix the dependency errors - it can be done but requires some kerfuffling​

Task for AI to do - the prompt, the testcase, whatever you want to call it. The goal is to make AI write the code and test until all errors are gone without intervention.

Create a new C file with a descriptive name. Inside it, implement a function to read the value for the CPU temperature using sysctlbyname with the modern path "dev.cpu.0.temperature". Note that FreeBSD returns this value as an integer in deci-Kelvin (e.g., 3000 means 300.0 Kelvin). Convert this value to Celsius by dividing by 10.0 and subtracting 273.15, then print it out in main() Also in the same code read the values for load average and print those out. If the compilation fails, intercept the errors and fix them.

The random models and how they did.

1)4.36Gqwen2.5-coder-7b-instruct-q4_k_m.ggufsuccess for load average but fail for temperature~ 54 t/s
2)6.87Ggemma-4-12b-Q4_K_M.gguffails caught in a reasoning loop~ 29 t/s
3)6.87Ggemma4-coding-Q4_K_M.ggufsuccess~ 30 t/s
4)7.95GLlama-3-Hercules-5.1-8B-Q8_0.gguffails to create the file~ 35 t/s
5)8.01GCodestral-22B-v0.1-IQ3_XXS.ggufsuccess~ 18 t/s
6)8.37Gqwen2.5-coder-14b-instruct-q4_k_m.ggufsuccess for temperature fail for load averages~ 30 t/s
7)8.43Gmicrosoft_Phi-4-reasoning-Q4_K_M.gguffails by hanging when fixing error~ 32 t/s
8)8.65GDevstral-Small-2-24B-Instruct-2512-UD-Q2_K_XL.ggufsuccess for temperature fails for load average~ 19 t/s
9)8.74GNorth-Mini-Code-1.0-UD-IQ1_M.ggufsuccess~ 54 t/s
a)8.91GQwen3.6-27B-UD-IQ2_XXS.gguffails~ 18 t/s
b)9.11Ggemma4-coding-Q6_K.ggufsuccess for temperature fails for load averages~ 25 t/s
c)9.11GQwen3.5-9B-Q8_0.ggufsuccess for temperature fails for load averages~ 25 t/s
d)14.84GQwen2.5-Coder-32B-Instruct-Q3_K_M.ggufsuccess for temperature fails slightly for load averages (2/3)~ 1 t/s

Warning:
If you run the below scripts you are responsible. You will see llama-cpp warnings like below so make sure you are safe.
srv llama_server: -----------------
srv llama_server: CORS is set to allow all origins ('*') and no API key is set
srv llama_server: this can be a security risk (cross-origin attacks)
srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
srv llama_server: -----------------
srv llama_server: -----------------
srv llama_server: the following feature(s) are enabled:
srv llama_server: server tools (experimental)
srv llama_server: do not expose the server to untrusted environments
srv llama_server: -----------------


Comments:
- llama-cpp was installed from ports first, that might have helped to build it from source
- for aider the dependencies that caused errors were installed as pkg and then edited out from requirements.txt
- the reason speed drops for models sized > NVRAM is that the CPU/RAM slows everything down when offloading from NVRAM is being done
- there is margin in the memory management in llama startup script to allow kde and chromium to run at the same time as the ai model
- there is a feeling that Vulkan not always is giving memory back after stopping llama-server with ctrl-c

Edit:
The script below has been tweaked and updated.

This script will run models up to ~19GB in size for any crap old hardware with 16GB RAM and 12GB NVRAM which means 32/34B models with Q4_K_M/Q3_K_M. The speed is estimated to around 0.5-1.5 t/s for any model that goes above 12GB. For smaller models speed is estimated to around 20-50 t/s.

Script to start llama and pick a model:

sh:
#!/bin/sh

MODELPATH="$HOME/AI-models"
TOTAL_VRAM=12.0      # Physical VRAM in GB
MAX_LOCKED_RAM=10.0  # FreeBSD limit for mlock in GB

# Path to your working Python 3.12 binary inside your venv
PY_BIN="$HOME/venvs/aider-env/bin/python3"

run_llama() {
    FILE_NAME="$1"
    FILE_SIZE_GB="$2"
    FULL_PATH="$MODELPATH/$FILE_NAME"

    echo "──────────────────────────────────────────────────────"
    echo "📊 Analyzing $FILE_NAME ($FILE_SIZE_GB GB)..."

    # Verify NVIDIA GPU and driver status cleanly
    echo "🎮 Verifying NVIDIA GPU status..."
    if [ ! -c /dev/nvidia0 ]; then
        echo "❌ Error: NVIDIA device node (/dev/nvidia0) is missing!"
        return 1
    fi

    NV_VERSION=$(sysctl -n hw.nvidia.version 2>/dev/null)
    if [ -z "$NV_VERSION" ]; then
        echo "❌ Error: NVIDIA kernel module is not active!"
        return 1
    fi
    echo "✅ GPU Status: NVIDIA Graphics Driver ($NV_VERSION) is active."

    echo "🧠 Reading model built-in metadata live..."

    export TARGET_GGUF_PATH="$FULL_PATH"

    METADATA=$("$PY_BIN" -c '
import sys, struct, os

def read_gguf_metadata(filepath):
    ctx_val, blocks_val = None, None
    try:
        with open(filepath, "rb") as f:
            # Verify Magic Header "GGUF"
            magic = f.read(4)
            if magic != b"GGUF": return None, None
        
            # Read Version (uint32) and V2/V3 fields
            version = struct.unpack("<I", f.read(4))[0]
            if version not in [2, 3]: return None, None
        
            # Read counts
            tensor_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
            kv_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
        
            # Types map according to GGUF spec
            # 0=uint8, 1=int8, 2=uint16, 3=int16, 4=uint32, 5=int32, 6=float32, 7=bool, 8=string, 9=array...
            for _ in range(kv_count):
                # Read key string
                key_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                key = f.read(key_len).decode("utf-8", errors="ignore")
            
                # Read value type
                val_type = struct.unpack("<I", f.read(4))[0]
            
                # Helper to skip or read values based on type
                def skip_value(t):
                    if t in [0, 1, 7]: f.seek(1, 1)
                    elif t in [2, 3]: f.seek(2, 1)
                    elif t in [4, 5, 6]: return struct.unpack("<I", f.read(4))[0]
                    elif t == 8: # String
                        s_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        f.seek(s_len, 1)
                    elif t == 9: # Array
                        arr_type = struct.unpack("<I", f.read(4))[0]
                        arr_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        for _ in range(arr_len): skip_value(arr_type)
                    elif t in [10, 11, 12, 13]: f.seek(8, 1) # uint64, int64, float64
                    return None

                res = skip_value(val_type)
            
                # Capture our specific targets cleanly
                if "context_length" in key and res:
                    ctx_val = res
                elif "block_count" in key and res:
                    blocks_val = res
                
                if ctx_val and blocks_val:
                    break
    except Exception:
        pass
    return ctx_val, blocks_val

ctx, blocks = read_gguf_metadata(os.environ.get("TARGET_GGUF_PATH", ""))
ctx_str = str(ctx) if ctx else ""
blocks_str = str(blocks) if blocks else ""
print(ctx_str + " " + blocks_str)
')

    TRAINED_CTX=$(echo "$METADATA" | awk '{print $1}')
    ACTUAL_LAYERS=$(echo "$METADATA" | awk '{print $2}')

    if [ -z "$TRAINED_CTX" ] || [ "$TRAINED_CTX" -lt 2048 ]; then
        TRAINED_CTX=32768
        echo "ℹ  Could not read trained context length. Using safe fallback: $TRAINED_CTX"
    else
        echo "🧠 Model is natively trained for a max of: $TRAINED_CTX tokens."
    fi

    if [ -z "$ACTUAL_LAYERS" ] || [ "$ACTUAL_LAYERS" -lt 10 ]; then
        ACTUAL_LAYERS=64
        echo "ℹ  Could not read layer count. Using baseline fallback: $ACTUAL_LAYERS layers."
    else
        echo "🧱 Model has an architectural structure of: $ACTUAL_LAYERS layers."
    fi

#
# MEMORY MANAGEMENT (DYNAMIC VRAM/RAM SPLIT & DYNAMIC KV CACHE)
#
    EVAL_OPTS=$(awk -v vram="$TOTAL_VRAM" -v ram="$MAX_LOCKED_RAM" -v size="$FILE_SIZE_GB" -v max_ctx="$TRAINED_CTX" -v total_layers="$ACTUAL_LAYERS" '
    BEGIN {
        system_vram_overhead = 1.8;
        available_vram = vram - system_vram_overhead;
    
        # SCENARIO 1: Model fits entirely in VRAM -> Use full quality f16 KV Cache
        if (size <= available_vram) {
            cache_type = "f16";
            cache_factor = 0.0065; # Full precision footprint
        
            vram_left_for_cache = available_vram - size;
            calculated_ctx = int((vram_left_for_cache * 1000) / (total_layers * cache_factor));
            ctx = int(calculated_ctx / 1024) * 1024;
        
            if (ctx < 2048) ctx = 2048;
            if (ctx > max_ctx) ctx = max_ctx;
        
            ngl = total_layers + 5;
            ram_fallback = 0;
            b = 512; ub = 256;
            mode = "mlock"; # Safe to lock in RAM since 0 model layers spill over
        }
        # SCENARIO 2: Model requires split-mode -> Switch to q4_0 KV Cache to save VRAM for layers
        else {
            cache_type = "q4_0";
            cache_factor = 0.0017; # Compressed footprint
        
            ctx = 4096;
            if (ctx > max_ctx) ctx = max_ctx;
        
            cache_vram_cost = (ctx / 1000) * total_layers * cache_factor;
            usable_vram_for_layers = available_vram - cache_vram_cost;
        
            if (usable_vram_for_layers < 0) usable_vram_for_layers = 0;
        
            vram_ratio = usable_vram_for_layers / size;
            ngl = int(total_layers * vram_ratio);
        
            if (ngl < 0) ngl = 0;
            if (ngl > (total_layers + 5)) ngl = total_layers + 5;
        
            ram_fallback = size - usable_vram_for_layers;
            b = 512; ub = 256;
            mode = "mmap"; # Use standard mmap to safely spill past FreeBSD mlock limits
        }
    
        if (ram_fallback > ram) {
            print "ERROR: Model requires " ram_fallback " GB system RAM offload, which exceeds your 10 GB limit!" > "/dev/stderr";
            printf "0 ERROR 0 0 0 error\n";
            exit 1;
        }
    
        printf "%d %s %d %d %d %s\n", ctx, cache_type, b, ub, ngl, mode;
    }')

    read -r ctx type b ub ngl mode <<EOF
$EVAL_OPTS
EOF

    if [ "$type" = "ERROR" ] || [ -z "$ctx" ] || [ "$ctx" -eq 0 ]; then
        echo "❌ ERROR: Launch Aborted! Constraints not met."
        return 1
    fi

    # Calculate safe prompt margin
    SAFE_PROMPT_LIMIT=$((ctx - 512))
    if [ "$SAFE_PROMPT_LIMIT" -lt 1024 ]; then
        SAFE_PROMPT_LIMIT=$ctx
    fi

    echo "⚙️  Optimized profile: CTX=$ctx, NGL=$ngl, BATCH=$b, UBATCH=$ub, CACHE=$type, MODE=$mode"
    echo "🔄 Context Shifting: Active (Infinite scrolling chat enabled)"
    echo "💡 Safe Prompt Limit: Do not exceed $SAFE_PROMPT_LIMIT tokens in a SINGLE prompt."
    echo "──────────────────────────────────────────────────────"

    # Start the server with q4_0 KV cache flags and flash attention
    llama-server \
        -t 3 \
        -b "$b" \
        -tb 4 \
        -ub "$ub" \
        -c "$ctx" \
        --context-shift \
        -fa on \
        --load-mode "$mode" \
        -ctk "$type" \
        -ctv "$type" \
        --jinja \
        --host 0.0.0.0 \
        --port 8080 \
        --alias kAI \
        --temp 0.4 \
        --top-p 0.1 \
        --min-p 0.05 \
        --tools all \
        -m "$FULL_PATH"
}

# Check if model directory exists
if [ ! -d "$MODELPATH" ]; then
    echo "❌ Error: Model directory $MODELPATH does not exist."
    exit 1
fi

echo "Pick an AI model to run: "

index=1
chars="123456789abcdefghijklmnopqrstuvwxyz"

MATCHED_MODELS=$(find "$MODELPATH" -maxdepth 1 -name "*.gguf" 2>/dev/null | while read -r path; do
    bytes=$(stat -f %z "$path")
    name=$(basename "$path")
    echo "$bytes $name"
done | sort -n)

old_ifs=$IFS
IFS='
'
for line in $MATCHED_MODELS; do
    IFS=$old_ifs
    bytes=$(echo "$line" | awk '{print $1}')
    name=$(echo "$line" | cut -d' ' -f2-)
    size_gb=$(awk -v b="$bytes" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
    selector=$(echo "$chars" | cut -c "$index")
    if [ -z "$selector" ]; then
        break
    fi
    eval "MENU_KEY_$selector=\"$name\""
    eval "MENU_SIZE_$selector=\"$size_gb\""
    printf "%s)   %6sG   %s\n" "$selector" "$size_gb" "$name"
    index=$((index + 1))
    IFS='
'
done
IFS=$old_ifs
echo "0) Exit"

printf "Your choice: "
read choice

if [ "$choice" = "0" ] || [ -z "$choice" ]; then
    echo "Exiting."
    exit 0
fi

eval "SELECTED_MODEL=\$MENU_KEY_$choice"
eval "SELECTED_SIZE=\$MENU_SIZE_$choice"

run_llama "$SELECTED_MODEL" "$SELECTED_SIZE"


/grandpa
FWIW, the lama_run.sh (my name) script fails if the models are stored in a directory tree with depth > 1.
Example:
Code:
tingo@locaal:~ $ find $HOME/.cache//huggingface/hub -name "*.gguf"
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q6_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/mmproj-Qwen_Qwen3.5-27B-bf16.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_M.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_S.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q2_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--TheDrummer_Orion-26B-A4B-v1.1-GGUF/snapshots/ecc1d958b6b4a83445ab80262965e151e00d97ac/TheDrummer_Orion-26B-A4B-v1.1-IQ3_XS.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--TheDrummer_Orion-26B-A4B-v1.1-GGUF/snapshots/ecc1d958b6b4a83445ab80262965e151e00d97ac/mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/125a02af4987b57c7deb88d7f2ec58a5725c07c0/Qwen3.8-27B-IQ2_XS.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/125a02af4987b57c7deb88d7f2ec58a5725c07c0/mmproj-Qwen3.8-27B-bf16.gguf
script, modified to use my paths, and I have taken out the 'maxdepth 1' from the find command
when run
Code:
tingo@locaal:~ $ sh bin/run_llama.sh
Pick an AI model to run: 
1)     0.00G   Qwen3.8-27B-IQ2_XS.gguf
2)     0.00G   Qwen_Qwen3.5-27B-Q2_K_L.gguf
3)     0.00G   Qwen_Qwen3.5-27B-Q3_K_L.gguf
4)     0.00G   Qwen_Qwen3.5-27B-Q3_K_M.gguf
5)     0.00G   Qwen_Qwen3.5-27B-Q3_K_S.gguf
6)     0.00G   Qwen_Qwen3.5-27B-Q6_K_L.gguf
7)     0.00G   TheDrummer_Orion-26B-A4B-v1.1-IQ3_XS.gguf
8)     0.00G   mmproj-Qwen3.8-27B-bf16.gguf
9)     0.00G   mmproj-Qwen_Qwen3.5-27B-bf16.gguf
a)     0.00G   mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf
0) Exit
Your choice: 1
──────────────────────────────────────────────────────
📊 Analyzing Qwen3.8-27B-IQ2_XS.gguf (0.00 GB)...
🎮 Verifying NVIDIA GPU status...
✅ GPU Status: NVIDIA Graphics Driver (NVIDIA UNIX x86_64 Kernel Module  595.84  Wed Jun 10 21:13:57 UTC 2026) is active.
🧠 Reading model built-in metadata live...
ℹ  Could not read trained context length. Using safe fallback: 32768
ℹ  Could not read layer count. Using baseline fallback: 64 layers.
⚙️  Optimized profile: CTX=32768, NGL=69, BATCH=512, UBATCH=256, CACHE=f16, MODE=mlock
🔄 Context Shifting: Active (Infinite scrolling chat enabled)
💡 Safe Prompt Limit: Do not exceed 32256 tokens in a SINGLE prompt.
──────────────────────────────────────────────────────
0.00.159.059 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.159.072 I device_info:
0.00.159.143 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
0.00.159.150 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
0.00.159.209 I system_info: n_threads = 3 (n_threads_batch = 4) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
0.00.159.215 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.159.238 I srv          init: running without SSL
0.00.159.270 I srv          init: using 11 threads for HTTP server
0.00.159.388 W srv  llama_server: -----------------
0.00.159.391 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.159.392 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.159.392 W srv  llama_server: -----------------
0.00.159.398 I srv         start: binding port with default address family
0.00.160.545 I srv  llama_server: loading model
0.00.160.552 I srv    load_model: loading model '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.629 I common_init_result: fitting params to device memory ...
0.00.160.631 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.160.664 E gguf_init_from_file: failed to open GGUF file '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf' (No such file or directory)
0.00.160.727 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf
0.00.160.737 E llama_model_load_from_file_impl: failed to load model
0.00.160.771 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.160.779 E gguf_init_from_file: failed to open GGUF file '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf' (No such file or directory)
0.00.160.800 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf
0.00.160.803 E llama_model_load_from_file_impl: failed to load model
0.00.160.804 E common_init_from_params: failed to load model '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.813 E srv    load_model: failed to load model, '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.814 I srv    operator(): operator(): cleaning up before exit...
0.00.161.110 E srv  llama_server: exiting due to model loading error
I'm too tired to fix it now. Maybe another night.
 
FWIW, the lama_run.sh (my name) script fails if the models are stored in a directory tree with depth > 1.

tingo@locaal:~ $ sh bin/run_llama.sh
Pick an AI model to run:
1) 0.00G Qwen3.8-27B-IQ2_XS.gguf
2) 0.00G Qwen_Qwen3.5-27B-Q2_K_L.gguf

ℹ Could not read trained context length. Using safe fallback: 32768
ℹ Could not read layer count. Using baseline fallback: 64 layers.

I'm too tired to fix it now. Maybe another night.

Yes, the scripts are particular to my environment and I see you are not getting the file sizes nor the gguf data.

The scripts have evolved to the ones in post #32, #33 where they are maybe just a little bit cleaner. Sorry for any inconvenience.

The later versions are more split up so the gguf reader is now a script of its own. If you want to run with Vulkan you will need to modify the command-to-run and the path to the llama-cpp executable in ai-launcher.sh.

In an attempt to simplify the scripts should be callable with parameters so you can test their output by calling them with actual values for the params.

/grandpa
 
So, while the linuxlator is installed (see #30) it's quite nice to be able to run the latest version of claude.

Talking to myself again.

Here's a technical aspect. It's very difficult to get claude to write to disk if you are low on NVRAM and RAM.

It seems claude wants to use - relatively speaking - large context windows and also there's a general feeling that claude veers towards the anthropic models.

Even with environment variables like --permission-mode bypassPermissions, as large context window as possible, running from a bash terminal and a .git subdirectory it's neigh on impossible to make claude create new files on local disk. It can - sometimes - compile source code and then create executables and it can read files, just not create new ones.

If anyone finds a solution to making claude create new files on systems with low RAM/NVRAM please let me know.

/grandpa
 
Here's a very crude and rough guide for how to pick an AI-model to run locally with purpose to get as much out of the model as possible.

mysterious quant in file namesomething about equivalent bitsnumber of parameters the model's been trained on (B)
BF16 / FP16161-8
Q8_0 / INT881-8
Q6_K / INT66.51-8
Q5_K_M5.51-8
Q5_K_S / Q5_051-14
GPTQ414-32
IQ4_NL4.514-32
Q4_K_M4.514-32
IQ4_XS4.2514-70
Q4_K_S / INT4432-70
IQ3_M3.570+
Q3_K_L 3.570+
IQ3_S / IQ3_XXS370-120
Q3_K_M3.370-120
IQ2 / Q2270+
IQ11.5100+

If you download a model with Q8_0 in the filename then look for a model that has 1B up to 8B in the filename. Or if you download a model with IQ2 then look for 70B and upwards.

/grandpa
 
Another aspect for stubborn old fools like me trying to run AI models on hardware that has parts from 2011 ...

Here' s a neat tool to catch-and-compress the context window before it's being sent to the model.

In effect you can use a context that is up to 3 times larger than the one llama-server has allocated in NVRAM or RAM without llama knowing.

This is the tool on github - sqz

Installation and setup (you will to have rust installed):
cargo install sqz-cli
sqz init --global


Claude should pick up this by automagic but for aider the script run-aider.sh in #33 needs to be changed:
sh:
# ==============================================================================
# 2. APPLY HARD OPTIMISATION AND MEMORY RESTRICTIONS (SQZ ADAPTIVE CHECK)
# ==============================================================================
export AIDER_MAP_TOKENS="0"
export AIDER_MAX_CHAT_HISTORY_TOKENS="8192"
export AIDER_TEMPERATURE="0.1"

# Check if sqz is installed
if [ -x "$HOME/.cargo/bin/sqz" ]; then
    SQZ_BIN="$HOME/.cargo/bin/sqz"
elif command -v sqz >/dev/null 2>&1; then
    SQZ_BIN="sqz"
else
    SQZ_BIN=""
fi
   
if [ -n "$SQZ_BIN" ]; then
    # SQZ found: Expand to virtual context window (3x)
    VIRTUAL_CTX=$(awk -v phys="$SELECTED_CTX" 'BEGIN {print int(phys * 3.0)}')
    export AIDER_MAX_CONTEXT_WINDOW="$VIRTUAL_CTX"
    USE_SQZ="true"
else
    # SQZ not found: Fallback to physical context window
    export AIDER_MAX_CONTEXT_WINDOW="$SELECTED_CTX"
    USE_SQZ="false"
fi
   
# ==============================================================================
# 5. EXECUTE THE NATIVE FREEBSD AIDER ENGINE FLOW WITH POSIX-SAFE SQZ PIPING
# ==============================================================================
echo "🤖 Initialising Aider Client Session..."
if [ "$USE_SQZ" = "true" ]; then
    echo "📊 Context Layer   : SQZ Intelligence Active [ojuschugh1/sqz]"
    echo "📊 Hardware Buffer : ${SELECTED_CTX} tokens | Virtual Window: ${AIDER_MAX_CONTEXT_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    
    # Running with sqz - saving lint output in /tmp and pipe to sqz then read back exit code to aider 
    "$AIDER_BIN_EXEC" \
     --lint-cmd "c: sh -c 'clang -fsyntax-only \"\$@\" > /tmp/aider_lint.log 2>&1; RC=\$?; cat /tmp/aider_lint.log | $SQZ_BIN; exit \$RC'" \
     --test-cmd "sh -c '$TEST_RUNNER > /tmp/aider_test.log 2>&1; RC=\$?; cat /tmp/aider_test.log | $SQZ_BIN; cat /tmp/aider_test.log; exit \$RC'" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
else
    echo "📊 Context Layer   : Native Hardware Direct (SQZ not found)"
    echo "📊 Allocated Buffer: ${AIDER_MAX_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    
    # No sqz found
    "$AIDER_BIN_EXEC" \
     --lint-cmd "c: clang -fsyntax-only" \
     --test-cmd "$TEST_RUNNER" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
fi

exit 0

You can see the output in aider:
.....
cpu_stats.c
Applied edit to cpu_stats.c
Commit 781fcd1 fix: implement cpu temp from sysctl and report system load averages
[sqz] 3/3 tokens (0% reduction)
....


Another way is to run sqz stats:
$ sqz stats

📊 sqz compression stats
──────────────────────────────────────────────────

0 tokens saved
↓ 0.0% average reduction

Compressions 2
Tokens in 7
Tokens out 7
Tokens saved 0
Avg reduction 0.0%

🗄 Cache
──────────────────────────────────────────────────
Entries 1
Size 804 B



/grandpa
 
Back
Top