Technical aspects of running local LLMs on FreeBSD

There is no FreeBSD-specific CUDA code, obviously.

we can agree on that
makes a change

As you said ports have to be patched

And there are some ports that havent been patched like Handbrake or Blender,
because you only have so much time, and as you said about Handbrake

FYI, there is no NVENC with Handbrake, because I didn't bother to submit the corresponding patches to the port.

That's correct isnt it.

So the project is addressing 2 issues

1) Ports that havent been patched to work with Cuda,
and installing the Linux version with Podman to get Cuda working.

2) Linux applications that require Cuda but cant be ported to Freebsd like Davinci Resolve because of licensing issues.

So if a port hasnt been patched the user has an alternative,
and can install the Linux version of the application with Podman and get it working with Cuda.

Rather than asking you or other devs to patch lots of different ports,
which you might not have the time for.

Surely thats a good thing from your point of view,
as it means less work patching.

Having pre configured applications that are easy to deploy,
and making things easier for user and giving them more choice is a good thing.

The end goal is getting an application working with Cuda for users,
we just have different ways of going about it.
 
ThisIsAGoodThread.png
 
To run a application/port that has been patched you have to prefix the command with nv-sglrun

for example to use ffmpeg with nvenc

Code:
nv-sglrun ffmpeg

script example

Code:
#!/bin/sh

#===============================================================================
# convert video to h265/aac
#===============================================================================


#===============================================================================
# script usage
#===============================================================================

usage()
{
# if argument passed to function echo it
[ -z "${1}" ] || echo "! ${1}"
# display help
echo "\
# convert video to h265/aac

$(basename "$0") -i input.mov -o output.mp4
-i infile.mov
-o outfile.mov :optional agument # if option not provided defaults to input-name.mp4"
exit 2
}


#===============================================================================
# error messages
#===============================================================================

NOTFILE_ERR='not a file'
INVALID_OPT_ERR='Invalid option:'
REQ_ARG_ERR='requires an argument'
WRONG_ARGS_ERR='wrong number of arguments passed to script'


#===============================================================================
# check number of aruments passed to script
#===============================================================================

[ $# -gt 0 ] || usage "${WRONG_ARGS_ERR}"


#===============================================================================
# getopts check options passed to script
#===============================================================================

while getopts ':i:o:h' opt
do
  case ${opt} in
     i) input="${OPTARG}"
    [ -f "${input}" ] || usage "${input} ${NOTFILE_ERR}";;
     o) output="${OPTARG}";;
     h) usage;;
     \?) usage "${INVALID_OPT_ERR} ${OPTARG}" 1>&2;;
     :) usage "${INVALID_OPT_ERR} ${OPTARG} ${REQ_ARG_ERR}" 1>&2;;
  esac
done
shift $((OPTIND-1))


#===============================================================================
# variables
#===============================================================================

input_nopath="${input##*/}"
input_name="${input_nopath%.*}"

# defaults for variables if not defined
output_default="${input_name}.mp4"


#===============================================================================
# functions
#===============================================================================

# h265 function
h265 () {
    nv-sglrun \
    ffmpeg \
    -hide_banner \
    -stats -v panic \
    -i "${input}" \
    -c:v hevc_nvenc \
    -pix_fmt p010le \
    -preset slow \
    -tier high \
    -rc vbr \
    -cq 22 \
    -b:v 0 \
    -maxrate 50M \
    -c:a aac \
    -b:a 320k \
    -ar 48000 \
    "${output:=${output_default}}"
}

# run the h265 function
h265 "${input}"

for gui applications you need to modify the desktop entry

for example to get obs studio working with nvenc

Code:
[i] Yes Master ? ls -l ~/.local/share/applications/com.obsproject.Studio.desktop
-rw-r--r--  1 djwilcox djwilcox 395  6 Jul 15:04 /home/djwilcox/.local/share/applications/com.obsproject.Studio.desktop

com.obsproject.Studio.desktop

Code:
[Desktop Entry]
Version=1.0
Name=OBS
GenericName=Streaming/Recording Software
Comment=Free and Open Source Streaming/Recording Software
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'
Icon=com.obsproject.Studio
Terminal=false
Type=Application
Categories=AudioVideo;Recorder;
StartupNotify=true
StartupWMClass=obs

notice the exec line

Code:
Exec=sh -c 'LD_LIBMAP="`nv-sglrun printenv LD_LIBMAP | grep -v libGL`" obs --websocket_ipv4_only'

as opposed to just running obs

Code:
Exec=obs --websocket_ipv4_only

However as i said if you try and install something like whisperx
which doesnt have a native Freebsd package with pip or conda with the Linuxulator or a Jail

Then it will fail to install because it will detect its running on Freebsd
and try and download Freebsd python wheels that dont exist

there are Freebsd python packages for torch

Code:
[i] Yes Master ? pkg search torch
py312-facenet-pytorch-2.5.3_4  Pretrained PyTorch face detection and recognition models
py312-lion-pytorch-0.2.4       PyTorch: Lion optimizer
py312-pytorch-2.12.1           PyTorch: Tensors and dynamic neural networks in Python
py312-pytorch-lightning-2.6.5  Lightweight PyTorch wrapper for ML researchers
py312-pytorchvideo-0.1.5_4     Video understanding deep learning library
py312-torch-geometric-2.8.0    Graph neural network library for PyTorch
py312-torchao-0.17.0           PyTorch: Package for applying ao techniques to GPU models
py312-torchaudio-2.11.0        PyTorch-based audio signal processing and machine learning library
py312-torchcodec-0.13.0        PyTorch media decoding and encoding
py312-torchdata-0.11.0         PyTorch: Composable data loading modules for PyTorch
py312-torchmetrics-1.9.0       PyTorch native metrics
py312-torchsde-0.2.6_2         SDE solvers and stochastic adjoint sensitivity analysis in PyTorch
py312-torchsummary-1.5.1_2     PyTorch: Model summary in PyTorch
py312-torchvision-0.27.0       PyTorch: Datasets, transforms and models specific to computer vision
pytorch-2.12.1                 Tensors and dynamic neural networks in Python (C++ library)
[

But they may not be the correct version for python application you are trying to install
and the application may require additional python libraries that dont have a Freebsd package

So the advantage of using Podman is you can set the container platform and os to Linux
which presents a Linux environment to applications like python so they download the python linux wheels

So for native Freebsd packages that have been patched to support nvenc for example
you can prefix the command with nv-sglrun or modify the desktop entry
 
BSD Jedi has some Ollama tutorial and a new one with a plug for some budget option cloud models.

I run 16GB vRAM nvidia on Debian Linux Ollama - rarely used, mainly as brain for Home Assistant....for fun.
I have good experience with Mixture of Experts models - gemma4. They really seem to use just parts of the model they actually need for given task.

Alas at work I have some good Ryzen AMD CPU, no vRAM to speak of but 64GB of DDR5 RAM (bought a year ago, yey) and it is surprisingly capable of running much bigger models than my home machine. Slow but who cares when crunching config files and not having conversations.

I also bought Google Coral TPUs on cheap but there is no chance I can make them work on naked FreeBSD - python libraries expect Linux. I was thinking of making a robot car with visual recognition on FreeBSD but all the python libraries are just made for Linux Rapsberry "drivers" too.

For more serious and private work 128GB is a must. But as OP pointed out if privacy is not a must, cloud models are better investment. Or rather 4,000 USD machine which is way dumber is just really bad investment right now.
 
Edit: 2026-09-26 - updated to make llama run by user

Some quick and crude steps to get llama-cpp (version: 0.4.0-dev (build 10931, commit 3057bb66c) running with CUDA in the Linuxlator on FreeBSD 15.1.


llama-cpp with CUDA on the Linuxlator

# ----------------------------------------------------------------
# The FreeBSD side
# ----------------------------------------------------------------
# The necessary filesystems
# Note: Last line only necessary if you have models already
# on the BSD side. Change it to your path.

/etc/fstab
proc /proc procfs rw 0 0
linprocfs /compat/linux/proc linprocfs rw 0 0
linsysfs /compat/linux/sys linsysfs rw 0 0
tmpfs /compat/linux/dev/shm tmpfs rw,late,mode=1777 0 0
/home/<insertyourusernamehere>/AI-models /compat/linux/home/<insertyourusernamehere>/AI-models nullfs rw,late 0 0

# /etc/rc.conf
sudo sysrc linux_enable="YES"
sudo sysrc kld_list+="nvidia-modeset nvidia-drm linux64"

# Note: these are the drivers tested
nvidia-driver-595.99.02 NVIDIA graphics driver userland
nvidia-drm-612-kmod-595.99.02.1501000 NVIDIA DRM Kernel Module
nvidia-drm-kmod-595.99.02 NVIDIA DRM kernel module
nvidia-kmod-595.99.02.1501000 NVIDIA graphics driver kernel module

# Setting up the Linuxlator
sudo mkdir -p /compat/linux
sudo debootstrap --arch=amd64 jammy /compat/linux http://archive.ubuntu.com/ubuntu
sudo mount /compat/linux/proc
sudo mount /compat/linux/sys
sudo mount /compat/linux/dev/shm
sudo service linux start

# Below is version 595.99.02
cd /usr/ports/x11/linux-nvidia-libs
sudo make install
sudo mkdir -p /compat/linux/home/<insertyourusernamehere>/AI-models
sudo mount -t nullfs /home/<insertyourusernamehere>/AI-models /compat/linux/home/<insertyourusernamehere>/AI-models

# ----------------------------------------------------------------
# The Linux side
# ----------------------------------------------------------------
# Initializing Ubuntu
sudo chroot /compat/linux /bin/bash
apt update
apt-get install -y software-properties-common && add-apt-repository universe && apt update
apt install -y python3-pip python3-dev wget curl build-essential
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX_3.4.30
apt install -y ninja-build cmake
mkdir -p /opt/llama/bin

# The FreeBSD uvm modification
mkdir /tmp/freebsd-cuda
cd /tmp/freebsd-cuda
git clone https://github.com/NapoleonWils0n/freebsd-cuda.git .
cd /opt/llama/bin
cp /tmp/freebsd-cuda/base-build/uvm_ioctl_override/uvm_ioctl_override.c .
gcc -shared -fPIC -o /opt/llama/bin/dummy-uvm.so /opt/llama/bin/uvm_ioctl_override.c

# Installing cuda stuff
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
# These three below might come with cuda-toolkit as well
apt install -y cuda-cudart-12-8
apt install -y libcublas-12-8
apt install -y libnccl2

# Builing llama-cpp
cd /opt
git clone https://github.com/ggerganov/llama.cpp.git llama-build
cd llama-build
apt install -y cuda-compiler-12-8 cuda-toolkit-12-8
export PATH="/usr/local/cuda-12.8/bin:$PATH"
echo 'export PATH="/usr/local/cuda-12.8/bin:$PATH"' >> /root/.bashrc
export CUDAToolkit_ROOT="/usr/local/cuda-12.8"
echo 'export CUDAToolkit_ROOT="/usr/local/cuda-12.8"' >> /root/.bashrc
# Fix a 32/64 bit library issue
rm -f /usr/lib/libcuda.so
ln -s /usr/local/cuda-12.8/targets/x86_64-linux/lib/stubs/libcuda.so /usr/lib/libcuda.so
cd /opt/llama-build
rm -rf build
cmake -B build -G Ninja -DGGML_CUDA=ON
cmake --build build --config Release --target llama-server


# Syncing user FreeBSD side/Linux side
grep "^insertyourusernamehere:" /etc/passwd | sudo tee -a /compat/linux/etc/passwd

# Running llama-cpp with cuda on FreeBSD
PATH="/compat/linux/usr/local/cuda-12.8/bin:$PATH" \
CUDAToolkit_ROOT="/compat/linux/usr/local/cuda-12.8" \
LD_LIBRARY_PATH
LD_PRELOAD="$DUMMY_UVM_HOOK" \

env LD_LIBRARY_PATH="/compat/linux/usr/local/cuda-12.8/lib64:/usr/lib64:/usr/lib/x86_64-linux-gnu" \
LD_PRELOAD="/compat/linux/opt/llama/bin/dummy-uvm.so" \
/compat/linux/opt/llama-build/build/bin/llama-server \
-m /home/<insertyourusernamehere>/AI-models/qwen2.5-coder-14b-instruct-q4_k_m.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 99


Note: Running nvidia-smi on the Linux side will not show CUDA - this is most likely because nvidia-smi not is built with the uvm modification.

Here's are some snippets of the output when llama-cpp start up:


0.00.014.956 I cmn common_param: common_params_print_info: build 10931 (3057bb66c) with GNU 11.2.0 for Linux x86_64
0.00.014.961 I cmn common_param: common_params_print_info: verbosity = 6 (adjust with the `-lv N` CLI arg)
0.00.014.961 I cmn common_param: device_info:
0.00.152.558 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3060 (12157 MiB, 11603 MiB free)
0.00.152.568 I cmn common_param: - CPU : Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz (16297 MiB, 16297 MiB free)
0.00.152.688 I cmn common_param: system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CUDA : ARCHS = 500,610,700,750,800,860,890,900,1200 | USE_GRAPHS = 1 | FA_QUANTS = q4_0-q4_0,q8_0-q8_0,f16-f16,bf16-bf16 | CPU : SSE3 = 1 | SSSE3 =
1 | AVX = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.152.695 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.152.825 I srv init: using 8 threads for HTTP server

....

0.00.264.148 I print_info: file size = 8.37 GiB (4.87 BPW)
0.00.264.669 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3060) (0000:01:00.0) - 11603 MiB free
0.00.357.956 D init_tokenizer: initializing tokenizer for type 2
0.00.392.019 I load: 0 unused tokens

....

0.00.515.627 I print_info: max token length = 256
0.00.515.872 I load_tensors: loading model tensors, this can take a while... (load_mode = none)
0.00.516.326 D load_tensors: layer 0 assigned to device CUDA0, is_swa = 0
0.00.516.327 D load_tensors: layer 1 assigned to device CUDA0, is_swa = 0
0.00.516.328 D load_tensors: layer 2 assigned to device CUDA0, is_swa = 0
0.00.516.328 D load_tensors: layer 3 assigned to device CUDA0, is_swa = 0


/grandpa
 
Last edited:
Some more technical aspects.

Running llama-cpp with CUDA on the Linuxlator and aider on the FreeBSD side with the same prompt (see #10) . Still on crap hardware.

The random models and how they did.

#SizeNameResultCUDAVULKAN
1)4.36Gqwen2.5-coder-7b-instruct-q4_k_m.ggufsuccess for load average but fail for temperature~ 62 t/s~ 54 t/s
2)
3)6.87Ggemma4-coding-Q4_K_M.ggufsuccess~ 33 t/s~ 30 t/s
4)
5)8.01GCodestral-22B-v0.1-IQ3_XXS.ggufsuccess~ 24 t/s~ 18 t/s
6)8.37Gqwen2.5-coder-14b-instruct-q4_k_m.ggufsuccess for temperature fail for load averages~ 32 t/s~ 30 t/s
7)
8)8.65GDevstral-Small-2-24B-Instruct-2512-UD-Q2_K_XL.ggufsuccess for temperature fails for load average~ 21 t/s~19 t/s
9)8.74GNorth-Mini-Code-1.0-UD-IQ1_M.ggufsuccess~ 72 t/s~ 54 t/s
a)
b)9.11Ggemma4-coding-Q6_K.ggufsuccess~ 27 t/s~25 t/s
c)9.11GQwen3.5-9B-Q8_0.gguffails stuck in a loop~ 34 t/s~25 t/s
d)14.84GQwen2.5-Coder-32B-Instruct-Q3_K_M.ggufsuccess for temperature fails slightly for load averages (2/3)~ 1.22 t/s~ 1 t/s

Notes:
-
Results may vary. Maybe it's the tweaking of the parameters. Especially suspect is the context window size.
- The feeling from #10 that Vulkan not always is giving memory back is totally gone. Switching between models does not cause out of device memory.
- In general it feels more smooth.

/grandpa
 
Edit 1: Sep 17 - all scripts have been (hopefully) simplified. This goes for #32 and #33.
Edit 2: Sep 20 - changes (for #32 and #33)
-
separated calculations for claude and aider since claude requires larger ctx
- basically hardcoded k and v cache to q4_0 to minimize VRAM utilization while keeping model sanity - kept option for aider for fb16 just for fun though
- added dynamic variables for llama flags -u and -ub
- added (probably unused) disabling of tools all for models with llama2 architecture
- tweaked llama flags
- tweaked overhead (1.20) to allow for kde/plasma, chromium, midnight commander and 4 konsole windows running at the same time as llama
- changed menu flow
- changed exit to stop llama server
- added test-runner.sh



Again with some technical aspects of local LLMs on FreeBSD. Sorry for scattered posts but I'm learning as I do this and there's alot of tweaking involved.

First a disclaimer.
- This post became so large I had to split it into 2 parts. see next post for the final script and some pictures.
- The scripts below are just a snapshot of the moment in time when this is being written. The evolution of llama-cpp and aider is going fast so expect change.
- The scripts uses sudo and llama-cpp runs as root (chroot) in the Linuxlator.
- The Scripts uses experimental flags like "tools all" for llama-cpp - be warned

This is the software from this snapshot in time:

  • aider 0.86.3.dev53+g5dc9490bb.d20260902 (pip installed in a venv and a modified requirements.txt)
  • FreeBSD 15.1-RELEASE-p3 releng/15.1-n283611-88e7371d9dc2 GENERIC amd64 including python3.12
  • linuxlator installed as described in #30
  • llama-cpp version: 0.4.0-dev (build 10931, commit 3057bb66c) installed as in #30
  • some AI-models from huggingface
What will these scripts do - briefly?
  • check computer RAM and NVRAM
  • read a list of models and their size from a specified directory
  • analyze those models for trained context window and layers
  • calculate optimal values for llama-cpp to fit inside the hardware constraints
  • show a list of models sorted with "fastest model first" and let user pick one to run
  • start aider with parameters coherent with llama-cpp
  • start claude with parameters coherent with llama-cpp
How do I run these scripts?

  1. Put the scripts in your shell path and make sure they are executable. Remember that user who runs needs to have sudo.
  2. run git-analyzer.sh
Note: A file name .git-analyzer.conf will be created and saved in user $HOME directory.

The scripts:

sh:
#!/bin/sh

# ==============================================================================
# 🌐 AI MASTER DASHBOARD & ORCHESTRATOR (Workspace-First Edition)
# ==============================================================================
CONFIG_FILE="$HOME/.git-analyzer.conf"

# Check if configuration profile is missing – if so, launch first-time setup!
if [ ! -f "$CONFIG_FILE" ]; then
    echo "⚠️  No configuration profile detected. Launching first-time setup..."
    "$HOME/scripts/config-paths.sh"
    if [ $? -ne 0 ]; then exit 1; fi
fi

# Load all saved environment variables directly into the shell
. "$CONFIG_FILE"

# Map to the internal variables expected by the rest of the script
SCRIPTS_DIR="$CONFIG_SCRIPTS_PATH"
MODELS_DIR="$CONFIG_MODELS_PATH"
WORKSPACE_DIR="$CONFIG_WORKSPACE_PATH"
AIDER_BIN_PATH="$CONFIG_AIDER_PATH"
CLAUDE_BIN_PATH="$CONFIG_CLAUDE_PATH"

SCRIPT_LAUNCHER="$SCRIPTS_DIR/ai-launcher.sh"

while true; do
echo "──────────────────────────────────────────────────────"
echo "🌐 AI Environment Master Dashboard"
echo "──────────────────────────────────────────────────────"
echo "💡 Active Workspace: $WORKSPACE_DIR"
echo "💡 Active AI Models: $MODELS_DIR"
echo "──────────────────────────────────────────────────────"
echo "Select execution path:"
echo "1) Run AI instantly using dynamic matrix optimiser"
echo "2) Change Configuration Paths"
echo "0) Exit"
echo "──────────────────────────────────────────────────────"
printf "Your choice: "
read -r menu_choice

case "$menu_choice" in

    1)
        echo "🚀 Invoking matrix pipeline..."
        cd "$WORKSPACE_DIR" || exit 1
       
        # Run launcher normally
        sh "$SCRIPT_LAUNCHER" "$SCRIPTS_DIR" "$MODELS_DIR" "$AIDER_BIN_PATH" "$CLAUDE_BIN_PATH"
       
        # 🔥 Catch exit code, if 10 close dashoboard
        if [ $? -eq 10 ]; then
            echo "Goodbye."
            exit 0
        fi
        ;;

    2)
        # Invoke the configuration script live
        "$HOME/scripts/config-paths.sh"
       
        # Immediately reload the new paths into memory to update the dashboard live
        if [ -f "$CONFIG_FILE" ]; then
            . "$CONFIG_FILE"
            SCRIPTS_DIR="$CONFIG_SCRIPTS_PATH"
            MODELS_DIR="$CONFIG_MODELS_PATH"
            WORKSPACE_DIR="$CONFIG_WORKSPACE_PATH"
            AIDER_BIN_PATH="$CONFIG_AIDER_PATH"
            CLAUDE_BIN_PATH="$CONFIG_CLAUDE_PATH"
            SCRIPT_LAUNCHER="$SCRIPTS_DIR/ai-launcher.sh"
        fi
        ;;

    0|*)
        echo "Exiting Dashboard. Goodbye."
        exit 0
        ;;
esac

done
sh:
#!/bin/sh

echo "──────────────────────────────────────────────────────"
echo "🖥️  FreeBSD 15 Hardware Architecture Discovery"
echo "──────────────────────────────────────────────────────"

# 1. DISCOVER SYSTEM RAM
PHYSMEM_BYTES=$(sysctl -n hw.physmem 2>/dev/null)

if [ -n "$PHYSMEM_BYTES" ]; then
    TOTAL_RAM_GB=$(awk -v b="$PHYSMEM_BYTES" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
    echo "🧠 System RAM Status:"
    echo "   • Total Physical RAM : $TOTAL_RAM_GB GB"
else
    echo "❌ Error: Unable to query physical system memory via sysctl."
    TOTAL_RAM_GB="0.00"
fi

echo "──────────────────────────────────────────────────────"

# 2. DISCOVER NVIDIA GRAPHICS VRAM
echo "🎮 NVIDIA Graphics VRAM Status:"

if [ ! -c /dev/nvidia0 ]; then
    echo "   ❌ Error: NVIDIA hardware device node (/dev/nvidia0) is missing!"
    TOTAL_VRAM_GB="0.00"
else
    NV_DRIVER_VERSION=$(sysctl -n hw.nvidia.version 2>/dev/null)
    if [ -n "$NV_DRIVER_VERSION" ]; then
        echo "   • Kernel Driver      : NVIDIA UNIX x86_64 ($NV_DRIVER_VERSION)"
    fi

    # Read the card model description string directly from the sysctl subsystem
    GPU_MODEL=$(sysctl -n hw.nvidia.0.description 2>/dev/null)
    if [ -n "$GPU_MODEL" ]; then
        echo "   • Hardware Model     : $GPU_MODEL"
    fi

    # MODERN FREEBSD 15 FALLBACK: Query pciconf for the active NVIDIA card memory map bounds
    # This reads the actual BAR memory windows allocated to the card on the PCIe bus
    VRAM_BYTES=$(pciconf -lv | grep -A 4 "vgapci" | grep -i "nvidia" -A 4 2>/dev/null | grep "bar.*vram" | sed -E 's/.*vram[[:space:]]*([0-9]+).*/\1/')
   
    if [ -z "$VRAM_BYTES" ]; then
        # Secondary fallback: Extract memory footprint sizing directly from the hardware description line
        VRAM_MB=$(sysctl -a | grep "nvidia" | grep -i "vram" | head -n 1 | awk '{print $NF}' | sed 's/[^0-9]//g')
    else
        VRAM_MB=$((VRAM_BYTES / 1024 / 1024))
    fi

    # If sysctl and pciconf both return empty, match against your known 12GB physical configuration
    if [ -z "$VRAM_MB" ] || [ "$VRAM_MB" -eq 0 ]; then
        VRAM_MB=12288
    fi

    TOTAL_VRAM_GB=$(awk -v mb="$VRAM_MB" 'BEGIN {printf "%.2f", mb / 1024}')
    echo "   • Dedicated VRAM     : $TOTAL_VRAM_GB GB ($VRAM_MB MB)"
fi

echo "──────────────────────────────────────────────────────"
echo "✅ Discovery phase complete."
echo "──────────────────────────────────────────────────────"

export INVENTORIED_RAM="$TOTAL_RAM_GB"
export INVENTORIED_VRAM="$TOTAL_VRAM_GB"
Python:
#!/usr/local/bin/python3.12
import sys
import struct

def read_gguf_metadata(filepath):
    ctx_val, blocks_val = None, None
    try:
        with open(filepath, "rb") as f:
            # 1. Verify Magic Header "GGUF"
            magic = f.read(4)
            if magic != b"GGUF":
                return None, None

            # 2. Read Version (uint32) - Extracts the integer out of the unpack tuple
            version = struct.unpack("<I", f.read(4))[0]
            if version not in set((2, 3)):
                return None, None
           
            # 3. Read metadata counts safely using version boundaries
            tensor_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
            kv_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]

            # 4. Iterate over key-value metadata blocks
            for _ in range(kv_count):
                key_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                key = f.read(key_len).decode("utf-8", errors="ignore")
                val_type = struct.unpack("<I", f.read(4))[0]
                   
                # Helper closure to skip or evaluate values based on GGUF type IDs
                def skip_value(t):
                    # 0=uint8, 1=int8, 7=bool
                    if t in set((0, 1, 7)):
                        f.seek(1, 1)
                    # 2=uint16, 3=int16
                    elif t in set((2, 3)):
                        f.seek(2, 1)
                    # 4=uint32, 5=int32, 6=float32
                    elif t in set((4, 5, 6)):
                        val = struct.unpack("<I", f.read(4))[0]
                        return val
                    # 8=string
                    elif t == 8:
                        s_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        f.seek(s_len, 1)
                    # 9=array
                    elif t == 9:
                        arr_type = struct.unpack("<I", f.read(4))[0]
                        arr_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        for _ in range(arr_len):
                            skip_value(arr_type)
                    # 10=uint64, 11=int64, 12=float64, 13=uint64, 14=int64
                    elif t in set((10, 11, 12, 13, 14)):
                        f.seek(8, 1)
                    return None
       
                res = skip_value(val_type)

                # 5. Capture context size and block layers out of matching keys
                if "context_length" in key and res is not None:
                    ctx_val = res
                elif "block_count" in key and res is not None:
                    blocks_val = res

                # Break loop immediately if both core targets are gathered
                if ctx_val is not None and blocks_val is not None:
                    break
                   
    except Exception:
        pass
       
    return ctx_val, blocks_val

if __name__ == "__main__":
    if len(sys.argv) > 1:
        # Pass index 1 to isolate the path argument string
        ctx, blocks = read_gguf_metadata(sys.argv[1])
        ctx_str = str(ctx) if ctx else "Unknown"
        blocks_str = str(blocks) if blocks else "Unknown"
        print(f"{ctx_str} {blocks_str}")
sh:
#!/bin/sh
# ==============================================================================
# ⚙️ AI CONFIGURATION LAYER ENVIRONMENT CONFIGURATOR
# ==============================================================================
CONFIG_FILE="$HOME/.git-analyzer.conf"
echo "──────────────────────────────────────────────────────"
echo "⚙️ AI Cluster Configuration Setup"
echo "──────────────────────────────────────────────────────"
# If file exists read what's in it
if [ -f "$CONFIG_FILE" ]; then
    . "$CONFIG_FILE"
fi
# 1. SCRIPTS PATH
printf "Enter full path to scripts directory [%s]: " "${CONFIG_SCRIPTS_PATH:-/home/user/scripts}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_SCRIPTS_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_SCRIPTS_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_SCRIPTS_PATH' does not exist."
    exit 1
fi
# 2. MODELS PATH
printf "Enter full path to GGUF models directory [%s]: " "${CONFIG_MODELS_PATH:-/home/user/AI-models}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_MODELS_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_MODELS_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_MODELS_PATH' does not exist."
    exit 1
fi
# 3. WORKSPACE PATH
printf "Enter full path to local development workspace [%s]: " "${CONFIG_WORKSPACE_PATH:-/home/user/Projects/ToobAmp}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_WORKSPACE_PATH=$(echo "$USER_INPUT" | sed 's|/$||')
fi
if [ ! -d "$CONFIG_WORKSPACE_PATH" ]; then
    echo "❌ Error: Directory '$CONFIG_WORKSPACE_PATH' does not exist."
    exit 1
fi
# 4. AIDER BINARY PATH
printf "Enter full path to Aider executable [%s]: " "${CONFIG_AIDER_PATH:-/home/user/venvs/aider-env/bin/aider}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_AIDER_PATH="$USER_INPUT"
fi
if [ ! -f "$CONFIG_AIDER_PATH" ]; then
    echo "❌ Error: Executable '$CONFIG_AIDER_PATH' does not exist."
    exit 1
fi
# 5. CLAUDE BINARY PATH
printf "Enter full path to Claude Code executable [%s]: " "${CONFIG_CLAUDE_PATH:-/home/user/.local/bin/claude}"
read -r USER_INPUT
if [ -n "$USER_INPUT" ]; then
    CONFIG_CLAUDE_PATH="$USER_INPUT"
fi
if [ ! -f "$CONFIG_CLAUDE_PATH" ]; then
    echo "❌ Error: Executable '$CONFIG_CLAUDE_PATH' does not exist."
    exit 1
fi
# Write out paths to config file
cat << EOC > "$CONFIG_FILE"
CONFIG_SCRIPTS_PATH="$CONFIG_SCRIPTS_PATH"
CONFIG_MODELS_PATH="$CONFIG_MODELS_PATH"
CONFIG_WORKSPACE_PATH="$CONFIG_WORKSPACE_PATH"
CONFIG_AIDER_PATH="$CONFIG_AIDER_PATH"
CONFIG_CLAUDE_PATH="$CONFIG_CLAUDE_PATH"
EOC
echo "──────────────────────────────────────────────────────"
echo "✅ Configuration layer compiled and saved to ~/.git-analyzer.conf"
echo "──────────────────────────────────────────────────────"
sh:
#!/bin/sh

# ==============================================================================
# 🧮 AI MASTER MATRIX & CONTEXT CALCULATORR
# ==============================================================================

SCRIPTS_DIR="$1"
MODEL_DIR="$2"
CLIENT_MODE="${3:-1}" # Default to 1 (Aider) if not passed

if [ -z "$SCRIPTS_DIR" ] || [ -z "$MODEL_DIR" ]; then
    echo "❌ Calculation Error: Missing critical orchestration path arguments." >&2
    exit 1
fi

SCRIPT_DISCOVERY="$SCRIPTS_DIR/hw-discovery.sh"
SCRIPT_READER="$SCRIPTS_DIR/gguf-reader.py"
PYTHON_BIN="python3.12"
SYSTEM_VRAM_OVERHEAD=1.20

if [ -f "$SCRIPT_DISCOVERY" ]; then
    . "$SCRIPT_DISCOVERY" >/dev/null 2>&1
else
    echo "❌ System Error: Hardware discovery module ($SCRIPT_DISCOVERY) not found!" >&2
    exit 1
fi

TOTAL_VRAM=${INVENTORIED_VRAM:-12.00}
AVAILABLE_VRAM=$(awk -v vram="$TOTAL_VRAM" -v ov="$SYSTEM_VRAM_OVERHEAD" 'BEGIN {printf "%.2f", vram - ov}')

MATCHED_MODELS=$(find "$MODEL_DIR" -maxdepth 1 -name "*.gguf" 2>/dev/null | sort)
if [ -z "$MATCHED_MODELS" ]; then
    echo "ℹ  No .gguf targets found inside $MODEL_DIR." >&2
    exit 0
fi

old_ifs=$IFS
IFS='
'
for model_path in $MATCHED_MODELS; do
    IFS=$old_ifs
    filename=$(basename "$model_path")
    bytes=$(stat -f %z "$model_path")
    size_gb=$(awk -v b="$bytes" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
   
    METADATA=$($PYTHON_BIN "$SCRIPT_READER" "$model_path" 2>/dev/null)
    trained_ctx=$(echo "$METADATA" | awk '{print $1}')
    layer_count=$(echo "$METADATA" | awk '{print $2}')
   
    if [ -z "$trained_ctx" ] || [ "$trained_ctx" = "Unknown" ]; then trained_ctx=32768; fi
    if [ -z "$layer_count" ] || [ "$layer_count" = "Unknown" ]; then layer_count=40; fi

    # Pass configuration factors live to the awk matrix builder
    awk -v av_vram="$AVAILABLE_VRAM" -v layers="$layer_count" -v m_size="$size_gb" -v max_trained="$trained_ctx" -v fname="$filename" -v mode_id="$CLIENT_MODE" '
    BEGIN {
        # ----------------------------------------------------------------------
        # CLAUDE FLOW (Menu Selection 2)
        # ----------------------------------------------------------------------
        if (mode_id ~ /2/) {
            cache_type = "q4_0";
            cache_factor = 0.0017;
            target_ctx = 65535;
            min_ctx = 32768;
           
            # Calculate VRAM cost for optimal 65k tokens in q4_0
            full_cache_cost = (target_ctx / 1000) * layers * cache_factor;
           
            # CLAUDE - SCENARIO A: Complete VRAM Fit (Model + Max Context)
            if (av_vram >= (m_size + full_cache_cost)) {
                ctx = target_ctx;
                ngl = layers + 5;
                mode = "mlock";
                speed_score = int(20000 - (m_size * 100));
            } else {
                # CLAUDE - SCENARIO B: System Spillover to System RAM (mmap)
                mode = "mmap";
               
                # Maintain a strict 1.00 GB hardware margin for system/driver stability
                vram_pool = av_vram - 1.00;
                if (vram_pool < 0) vram_pool = 0;
               
                # Calculate minimum context footprint cost (32k)
                min_cache_cost = (min_ctx / 1000) * layers * cache_factor;
               
                # Determine remaining VRAM allocatable to weights after securing 32k context
                vram_left_for_layers = vram_pool - min_cache_cost;
                if (vram_left_for_layers < 0) vram_left_for_layers = 0;
               
                # Scale layered offloading using an allocation dampening coefficient (0.85)
                vram_ratio = (vram_left_for_layers / m_size) * 0.85;
                ngl = int(layers * vram_ratio);
               
                if (ngl < 0) ngl = 0;
                if (ngl > layers) ngl = layers;
               
                # Dynamically expand context allocation beyond 32k utilizing remaining residual VRAM
                actual_model_cost = (ngl / layers) * m_size;
                vram_left_for_actual_cache = vram_pool - actual_model_cost;
                if (vram_left_for_actual_cache < 0) vram_left_for_actual_cache = 0;
               
                raw_ctx = (vram_left_for_actual_cache * 1000) / (layers * cache_factor);
                ctx = int(raw_ctx / 512) * 512;
               
                # Enforce architectural and runtime context limits
                if (ctx > target_ctx) ctx = target_ctx;
                if (ctx < min_ctx) ctx = min_ctx;
                if (ctx > max_trained) ctx = max_trained;
               
                speed_score = int(1000 * (ngl / layers));
            }
        }
        # ----------------------------------------------------------------------
        # AIDER FLOW (Menu Selection 1 / Standard Fallback)
        # ----------------------------------------------------------------------
        else {
            vram_left_for_cache = av_vram - m_size;
           
            if (vram_left_for_cache >= 0) {
                cache_type = "f16"; cache_factor = 0.0065;
                raw_ctx = (vram_left_for_cache * 1000) / (layers * cache_factor);
                ctx = int(raw_ctx / 512) * 512;
               
                if (ctx < 32768) {
                    cache_type = "q4_0"; cache_factor = 0.0017;
                    raw_ctx = (vram_left_for_cache * 1000) / (layers * cache_factor);
                    ctx = int(raw_ctx / 512) * 512;
                }
               
                if (ctx < 8192) ctx = 8192;
                if (ctx > max_trained) ctx = max_trained;
                ngl = layers + 5; mode = "mlock";
                speed_score = int(20000 - (m_size * 100));
            } else {
                cache_type = "q4_0"; cache_factor = 0.0017;
                ctx = 16384;
                if (ctx > max_trained) ctx = max_trained;

                cache_vram_cost = (ctx / 1000) * layers * cache_factor;
                usable_vram_for_layers = av_vram - cache_vram_cost;
                if (usable_vram_for_layers < 0) usable_vram_for_layers = 0;
               
                vram_ratio = usable_vram_for_layers / m_size;
                ngl = int(layers * vram_ratio); if (ngl < 0) ngl = 0;
                mode = "mmap";
                speed_score = int(1000 * (ngl / layers));
            }
        }
       
        if (ctx < 4096) ctx = 4096;
        cpu_layers = layers - ngl;
        if (cpu_layers < 0) cpu_layers = 0;
       
        # ----------------------------------------------------------------------
        # DYNAMIC BATCH PROFILING ENGINE (Calculated via RAM-Offload & Client Profile)
        # ----------------------------------------------------------------------
        if (cpu_layers == 0) {
            # Pure VRAM execution path - Maximum throughput profile
            hw_profile = "vram_full";
        } else if (cpu_layers > 0 && cpu_layers <= 20) {
            # Moderate RAM offloading - Hybrid processing profile
            hw_profile = "hybrid";
        } else {
            # Heavy RAM offloading - Interconnect bottleneck/PCIe mitigation profile
            hw_profile = "ram_heavy";
        }
       
        gsub(/ /, "___SPACE___", fname);
        # Emits profile as column index 8 (Shifting original raw filename to index 9)
        printf "%05d %s %d %d %d %s %s %s %s %s\n", speed_score, m_size, ngl, cpu_layers, ctx, cache_type, mode, hw_profile, fname, max_trained;
    }'

    IFS='
'
done
IFS=$old_ifs

There is one more script but dues to size restrictions it will be in next post.

/grandpa
 
Last edited:
Edit: Sep 17 - all scripts have been (hopefully) simplified. This goes for #32 and #33.
Edit 2: Sep 20 - changes (for #32 and #33)
-
separated calculations for claude and aider since claude requires larger ctx
- basically hardcoded k and v cache to q4_0 to minimize VRAM utilization while keeping model sanity - kept option for aider for fb16 just for fun though
- added dynamic variables for llama flags -u and -ub
- added (probably unused) disabling of tools all for models with llama2 architecture
- tweaked llama flags
- tweaked overhead (1.20) to allow for kde/plasma, chromium, midnight commander and 4 konsole windows running at the same time as llama
- changed menu flow
- changed exit to stop llama server
- added test-runner.sh


Continued from previous post.

Oh, I almost forgot. I changed the llama-cpp to log to /var/logs. This is required for that:
sudo touch /var/log/llama-server.log
sudo chown user /var/log/llama-server.log


sh:
#!/bin/sh

# ==============================================================================
# 🚀 AI INTERACTIVE INTERFACE & DAEMON LAUNCHER
# ==============================================================================

# Catch dynamic paths propagated from git-analyzer via parameters
SCRIPTS_DIR="$1"
MODEL_ROOT_DIR="$2"
AIDER_BIN="$3"
CLAUDE_BIN="$4"

if [ -z "$SCRIPTS_DIR" ] || [ -z "$MODEL_ROOT_DIR" ] || [ -z "$AIDER_BIN" ]; then
    echo "❌ Interface Error: Missing necessary environment parameter inputs. Aborting."
    exit 1
fi

# Set unified dynamic local script path bindings
SCRIPT_CALCULATOR="$SCRIPTS_DIR/ctx-calculator.sh"
TEST_RUNNER="$SCRIPTS_DIR/test-runner.sh"

# Static Host System Environment Settings
SYSTEM_LOG_FILE="/var/log/llama-server.log"

# Static Container & Linuxulator Architectural Nodes
CHROOT_PATH="/compat/linux"
CHROOT_ENV="/usr/bin/env"
LLAMA_SERVER_BIN="/opt/llama-build/build/bin/llama-server"
DUMMY_UVM_HOOK="/opt/llama/bin/dummy-uvm.so"

# Network Parameters
SERVER_HOST="0.0.0.0"
SERVER_PORT="8080"
AIDER_TARGET_HOST="127.0.0.1"

# 🛑 AUTOMATIC CLEANUP (Safe Trap Engine - clears background processes on break/exit)
cleanup_server() {
    if [ -n "$SERVER_PID" ]; then
        echo ""
        echo "Stopping background C++ server (PID: $SERVER_PID)..."
        sudo kill "$SERVER_PID" 2>/dev/null
        sudo pkill -9 -f "llama-server" 2>/dev/null
        sleep 0.5
    fi
}
trap cleanup_server EXIT INT TERM

# ==============================================================================
# 🆕 STEP 1: INTERACTIVE CLIENT INTERFACE SELECTION FORK
# ==============================================================================
echo "────────────────────────────────────────────────────────────────────────"
echo "Select your active AI coding interface:"
echo "1) Aider (Standard diff edit format toolchain)"
echo "2) Claude Code (Anthropic agent CLI toolchain)"
echo "────────────────────────────────────────────────────────────────────────"
printf "Your choice: "
read -r client_choice

if [ -z "$client_choice" ]; then
    client_choice="1"
fi

# FETCH MATRIX DATA PROPAGATING THE USER CHOICE SELECTION AS PARAMETER 3
RAW_DATA=$("$SCRIPT_CALCULATOR" "$SCRIPTS_DIR" "$MODEL_ROOT_DIR" "$client_choice")
if [ $? -ne 0 ] || [ -z "$RAW_DATA" ]; then
    echo "❌ Error: Matrix calculation pipeline failed."
    exit 1
fi

SORTED_MATRIX=$(echo "$RAW_DATA" | sort -k1,1r -n)

# RENDER INTERFACE HEADERS (Matching updated hardware matrix schema)
echo "──────────────────────────────────────────────────────────────────────────────────────────────────"
printf "%-6s  %-8s  %-7s  %-7s  %-6s  %-6s  %-5s  %-6s  %s\n" "INDEX" "SIZE" "VRAM-L" "RAM-L" "CTX" "CTXtr" "CACHE" "MODE" "OPTIMISED COMPATIBLE TARGET"
echo "──────────────────────────────────────────────────────────────────────────────────────────────────"

index=1
chars="123456789abcdefghijklmnopqrstuvwxyz"
old_ifs=$IFS
IFS='
'
MATRIX_DATABASE=""

for row in $SORTED_MATRIX; do
    IFS=$old_ifs
    size_gb=$(echo "$row" | awk '{print $2}')
    ngl=$(echo "$row" | awk '{print $3}')
    cpu_l=$(echo "$row" | awk '{print $4}')
    ctx=$(echo "$row" | awk '{print $5}')
    cache=$(echo "$row" | awk '{print $6}')
    mode=$(echo "$row" | awk '{print $7}')
    hw_prof=$(echo "$row" | awk '{print $8}')     # <-- CAPTURED FROM INDEX 8
    raw_filename=$(echo "$row" | awk '{print $9}')  # <-- SHIFTED TO INDEX 9
    ctxtr=$(echo "$row" | awk '{print $10}')       # <-- SHIFTED TO INDEX 10
    filename=$(echo "$raw_filename" | sed 's/___SPACE___/ /g')

    selector=$(echo "$chars" | cut -c "$index")
    if [ -z "$selector" ]; then selector="-"; fi

    # Bind structural profile data to relational cache record (HW profile at index 7)
    MATRIX_DATABASE="${MATRIX_DATABASE}${selector}|${filename}|${ctx}|${ngl}|${mode}|${cache}|${hw_prof}
"

    printf "%-6s  %-8s  %-7s  %-7s  %-6s  %-6s  %-5s  %-6s  %s\n" "${selector})" "${size_gb}G" "$ngl" "$cpu_l" "$ctx" "$ctxtr" "$cache" "$mode" "$filename"
    index=$((index + 1))
    IFS='
'
done
IFS=$old_ifs

echo "0)      Exit"
echo "──────────────────────────────────────────────────────────────────────────────────────────────────"
printf "Your choice: "
read -r choice

if [ "$choice" = "0" ] || [ -z "$choice" ]; then
    echo "Exiting."
    exit 0
fi

SELECTED_ROW=$(echo "$MATRIX_DATABASE" | grep "^${choice}|" | head -n 1)

if [ -z "$SELECTED_ROW" ]; then
    echo "❌ Input Error: Invalid matrix selection."
    exit 1
fi

SELECTED_MODEL=$(echo "$SELECTED_ROW" | cut -d'|' -f2)
SELECTED_CTX=$(echo "$SELECTED_ROW" | cut -d'|' -f3)
SELECTED_NGL=$(echo "$SELECTED_ROW" | cut -d'|' -f4)
SELECTED_MODE=$(echo "$SELECTED_ROW" | cut -d'|' -f5)
SELECTED_CACHE=$(echo "$SELECTED_ROW" | cut -d'|' -f6)
SELECTED_PROF=$(echo "$SELECTED_ROW" | cut -d'|' -f7) # <-- PARSED HARDWARE COMPUTE PROFILE

# ----------------------------------------------------------------------
# HIGH-OPTIMISED BATCH CONFIGURATION MAPPING VIA COMPUTE PROFILE
# ----------------------------------------------------------------------
case "$SELECTED_PROF" in
    vram_full)
        # Weights reside 100% in VRAM. Max out prefill speeds for modern architectures (Qwen/Granite).
        CALCULATED_B=2048
        CALCULATED_UB=1024
        ;;
    hybrid)
        # Moderate RAM overflow. Balance prompt processing speed against PCIe bus limitations.
        CALCULATED_B=512
        CALCULATED_UB=128
        ;;
    ram_heavy|*)
        # Heavy system RAM execution (~20GB+ models). Protect system memory from freezing.
        CALCULATED_B=256
        CALCULATED_UB=64
        ;;
esac


# ==============================================================================
# 🛠️ DYNAMIC ARCHITECTURAL ROUTER (Native Templates & Tooling Capabilities)
# ==============================================================================
MODEL_LOWER=$(echo "$SELECTED_MODEL" | tr '[:upper:]' '[:lower:]')
TOOLS_SUPPORTED="true" # Default to true for modern models

if echo "$MODEL_LOWER" | grep -iq "llama-3" || echo "$MODEL_LOWER" | grep -iq "llama3"; then
    TEMPLATE_FLAG="--chat-template llama3"
elif echo "$MODEL_LOWER" | grep -iq "llama-4" || echo "$MODEL_LOWER" | grep -iq "llama4"; then
    TEMPLATE_FLAG="--chat-template llama3" # Future-proof for newer structures
elif echo "$MODEL_LOWER" | grep -iq "llama" || echo "$MODEL_LOWER" | grep -iq "codellama"; then
    # Legacy Llama 2 architecture detected
    TEMPLATE_FLAG="--chat-template llama2"
    TOOLS_SUPPORTED="false"
elif echo "$MODEL_LOWER" | grep -iq "qwen"; then
    TEMPLATE_FLAG="--chat-template chatml"
elif echo "$MODEL_LOWER" | grep -iq "gemma"; then
    TEMPLATE_FLAG="--chat-template gemma"
else
    # Granite and modern generic fallback structures
    TEMPLATE_FLAG="--chat-template chatml"
fi

# Apply the strict tooling toggle based on structural capabilities
if [ "$TOOLS_SUPPORTED" = "true" ]; then
    LLAMA_TOOLS="--tools all"
    echo "📊 Tool Strategy   : Autonomous Mode (Native Agent Tooling Enabled)"
else
    LLAMA_TOOLS=""
    echo "📊 Tool Strategy   : Legacy Mode (Native Tooling Disabled for Llama 2 Structure)"
fi

# 2. Synchronise the template directly into your already mounted model directory
cp "$HOST_TEMPLATE" "$CHROOT_TEMPLATE_PATH"
chmod 644 "$CHROOT_TEMPLATE_PATH"

echo "────────────────────────────────────────────────────────────────────────"
echo "🚀 Launching C++ CUDA Server silently in the background (LAN Active)..."
echo "📊 Batch Config    : -b $CALCULATED_B | -ub $CALCULATED_UB (Profile: $SELECTED_PROF)"
echo "────────────────────────────────────────────────────────────────────────"

truncate -s 0 "$SYSTEM_LOG_FILE"

#        --jinja --chat-template-file "$MODEL_ROOT_DIR/$TARGET_TEMPLATE_NAME" \
#        -r "<|im_im_end|>" \

# INVOKE HEADLESS DAEMON (Dynamically optimized batch sizes running with custom agnostic template)
sudo chroot "$CHROOT_PATH" "$CHROOT_ENV" \
    PATH="/usr/local/cuda-12.8/bin:$PATH" \
    CUDAToolkit_ROOT="/usr/local/cuda-12.8" \
    LD_LIBRARY_PATH="/usr/local/cuda-12.8/lib64:/usr/lib64:/usr/lib/x86_64-linux-gnu" \
    LD_PRELOAD="$DUMMY_UVM_HOOK" \
    "$LLAMA_SERVER_BIN" \
        -t 3 -tb 4 -b "$CALCULATED_B" -ub "$CALCULATED_UB" --parallel 1 -fit off \
        -n -1 --repeat-penalty 1.15 --repeat-last-n 512 --temp 0.3 -fa on \
    --top-k 20 --top-p 0.95 $TEMPLATE_FLAG \
        --ctx-size "$SELECTED_CTX" \
        --n-gpu-layers "$SELECTED_NGL" \
        --load-mode "$SELECTED_MODE" \
        -ctk "$SELECTED_CACHE" \
        -ctv "$SELECTED_CACHE" \
        -r "<|end|>" -r "</tool_call>" -r "<|end_of_turn|>" -r " </turn>" \
        --host "$SERVER_HOST" --port "$SERVER_PORT" $LLAMA_TOOLS \
        -m "$MODEL_ROOT_DIR/$SELECTED_MODEL" > "$SYSTEM_LOG_FILE" 2>&1 &

SERVER_PID=$!

# HANDSHAKE NETWORK PORT MONITOR (Robust router initialization check via /v1/models)
echo "⏳ Streaming model tracks into physical memory boundaries..."
ready=0
sleep 1.5
for i in $(seq 1 120); do
    if curl -s "http://127.0.0.1:$SERVER_PORT/v1/models" | grep -q "$SELECTED_MODEL" 2>/dev/null; then
        echo ""
        echo "✅ Model layers fully streamed and indexed!"
        ready=1
        break
    fi
    printf "."
    sleep 1
done
printf "\n"

if [ $ready -eq 0 ]; then
    echo "❌ Error: Server initialization network handshake timed out."
    exit 1
fi

echo "✅ Server network handshake verified. Listening on port $SERVER_PORT."
echo "────────────────────────────────────────────────────────────────────────"

CURRENT_TIME_STRING=$(date "+%A, %d %B %Y at %H:%M %Z")

# Set time and basic operational rules
cat << EOF > CLAUDE.md
# FreeBSD AI Cluster Environment Instructions
- Host Operating System: FreeBSD 15 (CUDA Hardware Accelerated)
- Current Real-World Timestamp: $CURRENT_TIME_STRING

## Operational Rules (Strict)
- Answer the user's direct question accurately and concisely using the timestamp above.
- STOP generating text immediately after answering. Do not speculate, suggest next steps, or provide structural code unless explicitly asked to do so.
- Keep your output minimal and yield back control to the prompt.
EOF

case "$client_choice" in
    2)
        SCRIPT_CLAUDE="$SCRIPTS_DIR/run-claude.sh"
        if [ -f "$SCRIPT_CLAUDE" ]; then
            sh "$SCRIPT_CLAUDE" "$SELECTED_MODEL" "$SELECTED_CTX" "$CLAUDE_BIN"
            echo "👋 Claude Code session closed. Terminating AI Cluster Environment..."
            sleep 0.5
            exit 10
        else
            echo "❌ System Error: Claude runner script ($SCRIPT_CLAUDE) is missing!"
            exit 1
        fi
        ;;

    1|*)
        SCRIPT_AIDER="$SCRIPTS_DIR/run-aider.sh"
        if [ -f "$SCRIPT_AIDER" ]; then
            sh "$SCRIPT_AIDER" "$SELECTED_MODEL" "$SELECTED_CTX" "$AIDER_BIN"
            echo "👋 Aider session closed. Terminating AI Cluster Environment..."
            sleep 0.5
            exit 10
        else
            echo "❌ System Error: Aider runner script ($SCRIPT_AIDER) is missing!"
            exit 1
        fi
        ;;
esac
sh:
#!/bin/sh

# ==============================================================================
# 🚀 FREEBSD NATIVE HARDWARE-ACCELERATED AIDER CLIENT RUNNER (RAM-Engine Edition)
# ==============================================================================
SELECTED_MODEL="$1"
SELECTED_CTX="$2"
AIDER_BIN_EXEC="$3"

if [ -z "$SELECTED_MODEL" ] || [ -z "$SELECTED_CTX" ] || [ -z "$AIDER_BIN_EXEC" ]; then
    echo "❌ Aider Pipeline Error: Missing critical orchestration parameters. Aborting."
    exit 1
fi

# Local Script Path Resolution Nodes
SCRIPTS_DIR="$HOME/scripts"
TEST_RUNNER="$SCRIPTS_DIR/test-runner.sh"
AIDER_TARGET_HOST="127.0.0.1"
SERVER_PORT="8080"

# 1. MAP CORE API ROUTING AND MODEL SELECTION LAYERS
export OPENAI_API_BASE="http://${AIDER_TARGET_HOST}:${SERVER_PORT}/v1"
export OPENAI_API_KEY="dummy"
export AIDER_MODEL="openai/$SELECTED_MODEL"

# 🔥 FORCE CONCISE RESPONSES AND STOP OVER-REASONING:
export AIDER_TEMPERATURE="0.1"

# ==============================================================================
# 2. APPLY HARD OPTIMISATION AND MEMORY RESTRICTIONS (SQZ ADAPTIVE CHECK)
# ==============================================================================
export AIDER_MAP_TOKENS="0"
export AIDER_MAX_CHAT_HISTORY_TOKENS="4096"

# Check if sqz is installed
if [ -x "$HOME/.cargo/bin/sqz" ]; then
    SQZ_BIN="$HOME/.cargo/bin/sqz"
elif command -v sqz >/dev/null 2>&1; then
    SQZ_BIN="sqz"
else
    SQZ_BIN=""
fi

if [ -n "$SQZ_BIN" ]; then
    # SQZ found: Expand to virtual context window (3x)
    VIRTUAL_CTX=$(awk -v phys="$SELECTED_CTX" 'BEGIN {print int(phys * 3.0)}')
    export AIDER_MAX_CONTEXT_WINDOW="$VIRTUAL_CTX"
    USE_SQZ="true"
else
    # SQZ not found: Fallback to physical context window
    export AIDER_MAX_CONTEXT_WINDOW="$SELECTED_CTX"
    USE_SQZ="false"
fi

# ==============================================================================
# 3. DYNAMIC METADATA GENERATION FOR UNKNOWN LOCAL MODELS
# ==============================================================================
METADATA_FILE="/tmp/aider_model_metadata_${SELECTED_MODEL}.json"
cat << EOF > "$METADATA_FILE"
{
  "openai/${SELECTED_MODEL}": {
    "max_context_tokens": ${AIDER_MAX_CONTEXT_WINDOW},
    "max_output_tokens": 4096,
    "input_cost_per_token": 0.0,
    "output_cost_per_token": 0.0
  }
}
EOF
export AIDER_MODEL_METADATA_FILE="$METADATA_FILE"

# ==============================================================================
# 5. EXECUTE THE NATIVE FREEBSD AIDER ENGINE FLOW WITH POSIX-SAFE SQZ PIPING
# ==============================================================================
echo "🤖 Initialising Aider Client Session..."
if [ "$USE_SQZ" = "true" ]; then
    echo "📊 Context Layer   : SQZ Intelligence Active [ojuschugh1/sqz]"
    echo "📊 Hardware Buffer : ${SELECTED_CTX} tokens | Virtual Window: ${AIDER_MAX_CONTEXT_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    # With sqz
    "$AIDER_BIN_EXEC" \
     --model "openai/${SELECTED_MODEL}" \
     --lint-cmd "c: sh -c 'clang -fsyntax-only \"\$@\" > /tmp/aider_lint.log 2>&1; RC=\$?; cat /tmp/aider_lint.log | $SQZ_BIN; exit \$RC'" \
     --test-cmd "sh -c '$TEST_RUNNER > /tmp/aider_test.log 2>&1; RC=\$?; cat /tmp/aider_test.log | $SQZ_BIN; cat /tmp/aider_test.log; exit \$RC'" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
else
    echo "📊 Context Layer   : Native Hardware Direct (SQZ not found)"
    echo "📊 Allocated Buffer: ${AIDER_MAX_CONTEXT_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    
    # Without sqz
    "$AIDER_BIN_EXEC" \
     --model "openai/${SELECTED_MODEL}" \
     --lint-cmd "c: clang -fsyntax-only" \
     --test-cmd "$TEST_RUNNER" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
fi

# Clean up temp JSON file
rm -f "$METADATA_FILE"
exit 0
#!/bin/sh

# ==============================================================================
# 🤖 CLAUDE CODE CLI RUNNER
# ==============================================================================
# Purpose: Dynamically maps Anthropic environmental layers to perfectly match
# the hardware and token boundaries calculated by the matrix engine.
# Utilises full parameter paths and aggressive context compression.
# ==============================================================================

# Inherit dynamic runtime metrics forwarded from the parent launcher pipeline
SELECTED_MODEL="$1"
SELECTED_CTX="$2"
CLAUDE_BIN_PATH="$3"

if [ -z "$SELECTED_MODEL" ] || [ -z "$SELECTED_CTX" ] || [ -z "$CLAUDE_BIN_PATH" ]; then
echo "❌ Execution Error: Missing dynamic context or executable matrix parameters." >&2
exit 1
fi

# 1. DYNAMICALLY COMPUTE CLAUDE BOUNDARIES BASED ON ENGINE CALCULATIONS
# We calculate the safe compaction threshold at exactly 80% of the active server context
# to provide an aggressive buffer against heavy initial Git repository index states.
COMPACT_WINDOW=$(awk -v ctx="$SELECTED_CTX" 'BEGIN {printf "%d", ctx * 0.80}')

# ==============================================================================
# 2. MAP THE ADAPTIVE ANTHROPIC AGENT ENVIRONMENT LAYERS
# ==============================================================================
export ANTHROPIC_MODEL="claude-sonnet-5"
export ANTHROPIC_BASE_URL="http://127.0.0.1:8080"
export ANTHROPIC_DEFAULT_SONNET_MODEL="claude-sonnet-5"
export ANTHROPIC_DEFAULT_OPUS_MODEL="claude-sonnet-5"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-sonnet-5"
export ANTHROPIC_AUTH_TOKEN="ollama"
export ANTHROPIC_SMALL_FAST_MODEL="claude-sonnet-5"

# Apply the dynamic context window settings directly to Claude Code's memory engine
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="$COMPACT_WINDOW"
export CLAUDE_AUTOCOMPACT_PCT_OVERRIDE="80"
export CLAUDE_CODE_DISABLE_1M_CONTEXT="1"
export CLAUDE_CODE_MAX_OUTPUT_TOKENS="2048"

# 🔥 THE TRUE LOCAL LOCAL-MODEL FIX:
# Strictly disable cloud-dependent fast-modes to ensure Claude Code
# parses local native JSON tool schemas instead of hallucinating.
export CLAUDE_CODE_DISABLE_FAST_MODE="1"
export CLAUDE_CODE_MAX_CHAT_HISTORY_TOKENS="2048"
export CLAUDE_CODE_DISABLE_GIT_INDEXING="1"

# Set Temperature to low precision to guarantee perfect syntax formatting during edits
export CLAUDE_CODE_MAX_RETRIES="5"
export DISABLE_PROMPT_CACHING="1"
export CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING="1"
export MAX_THINKING_TOKENS="0"
export DISABLE_INTERLEAVED_THINKING="1"
export CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY="1"
export DISABLE_TELEMETRY="1"
export CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY="1"
export ENABLE_TOOL_SEARCH="auto"
export CLAUDE_CODE_GIT_BASH_PATH="/usr/local/bin/bash"

# 🔥 FIXES THE CHANNEL SPINNING LOOP & SILENT TIMEOUTS:
export CLAUDE_CODE_DISABLE_CHANNELS="1"
export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS="1"
export MCP_PROTOCOL_NEGOTIATION="1"
export MCP_TOOL_TIMEOUT="60000"
export CLAUDE_CODE_DISABLE_JSON_SCHEMA_VALIDATION="1"
export CLAUDE_CODE_ATTRIBUTION_HEADER="0"
#export CLAUDE_CODE_DISABLE_FILE_WRITER="1"


echo ""
echo "--- Llama.cpp Configuration Loaded ---"
echo "Model Selected : $SELECTED_MODEL (Mapped to Sonnet-5 Profile)"
echo "LAN Endpoint : $ANTHROPIC_BASE_URL"
echo "Calculated CTX : $SELECTED_CTX tokens"
echo "Compact Buffer : $COMPACT_WINDOW tokens (Triggers auto-rolling shift at 80%)"
echo "Executable : $CLAUDE_BIN_PATH"
echo "────────────────────────────────────────────────────────────────────────"

# 3. FIXED: Fire up Claude Code natively utilizing the dynamic variable path parameter!
"$CLAUDE_BIN_PATH" --permission-mode bypassPermissions \
--mcp-config "$HOME/.claude.json"

sh:
sh:
#!/bin/sh
# test_runner.sh

# 1. Automatically grab ALL .c files present in the current directory
ALL_FILES=$(ls *.c 2>/dev/null)

if [ -n "$ALL_FILES" ]; then
    # 2. Pick a name for the binary based on the most recently modified C file
    MAIN_FILE=$(git diff-tree --no-commit-id --name-only -r HEAD | grep "\.c$" | head -n 1)
    if [ -z "$MAIN_FILE" ]; then
        MAIN_FILE=$(ls -t *.c | head -n 1)
    fi
    BINARY="${MAIN_FILE%.c}"
   
    echo "⚙️ Automatically compiling all files ($ALL_FILES) into binary [$BINARY] using clang..."
   
    # 3. Compile every single C file together
    clang $ALL_FILES -o "$BINARY" && ./"$BINARY"
else
    echo "❌ Error: Aider test runner couldn't locate any active C files."
    exit 1
fi


/grandpa
 
Last edited:
So, while the linuxlator is installed (see #30) it's quite nice to be able to run the latest version of claude.

This can be done by doing 3 steps providing you have node installed.
  1. sudo npm install -g --allow-scripts=@anthropic-ai/claude-code @anthropic-ai/claude-code
  2. sudo mkdir -p /lib/x86_64-linux-gnu
  3. sudo ln -sf /compat/linux/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
What this does is that node grabs the latest version of claude and installs it to user profile. That version of claude has a dependency to a linux library. Since we have the linuxlator we have that library. Create the folder and symlink to trick claude into running.


/grandpa
 
Last edited:
Now, the fun begins.

sudo pkg install mcp-server-filesystem


And once claude is running
claude mcp add filesystem -- /usr/local/bin/mcp-server-filesystem /home/user/Projects
What MCP tools do you have access to right now?


For even more fun search github for mcp-servers.

/grandpa
 
Great topic for a thread, thanks cracauer@.

Here's some technical aspects for running AI models locally on unbalanced crap hardware with llama-cpp, some random AI models from huggingface and aider.

The crap Hardware:

CPU Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz ~ 2011​
16 Gb DD3 RAM @ 1600 ~ 2007​
RTX 3060 with 12 Gb NVRAM ~ 2021​

Installations:

llama-cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake -DGGML_VULKAN=1 ..
cmake --build . --config Release -j 3
cp llama* lib*.so* ~/.local/bin/


aider - this one is a bit trickier - git clone the source and then pip install in a venv and fix the dependency errors - it can be done but requires some kerfuffling​

Task for AI to do - the prompt, the testcase, whatever you want to call it. The goal is to make AI write the code and test until all errors are gone without intervention.

Create a new C file with a descriptive name. Inside it, implement a function to read the value for the CPU temperature using sysctlbyname with the modern path "dev.cpu.0.temperature". Note that FreeBSD returns this value as an integer in deci-Kelvin (e.g., 3000 means 300.0 Kelvin). Convert this value to Celsius by dividing by 10.0 and subtracting 273.15, then print it out in main() Also in the same code read the values for load average and print those out. If the compilation fails, intercept the errors and fix them.​

The random models and how they did.

1)4.36Gqwen2.5-coder-7b-instruct-q4_k_m.ggufsuccess for load average but fail for temperature~ 54 t/s
2)6.87Ggemma-4-12b-Q4_K_M.gguffails caught in a reasoning loop~ 29 t/s
3)6.87Ggemma4-coding-Q4_K_M.ggufsuccess~ 30 t/s
4)7.95GLlama-3-Hercules-5.1-8B-Q8_0.gguffails to create the file~ 35 t/s
5)8.01GCodestral-22B-v0.1-IQ3_XXS.ggufsuccess~ 18 t/s
6)8.37Gqwen2.5-coder-14b-instruct-q4_k_m.ggufsuccess for temperature fail for load averages~ 30 t/s
7)8.43Gmicrosoft_Phi-4-reasoning-Q4_K_M.gguffails by hanging when fixing error~ 32 t/s
8)8.65GDevstral-Small-2-24B-Instruct-2512-UD-Q2_K_XL.ggufsuccess for temperature fails for load average~ 19 t/s
9)8.74GNorth-Mini-Code-1.0-UD-IQ1_M.ggufsuccess~ 54 t/s
a)8.91GQwen3.6-27B-UD-IQ2_XXS.gguffails~ 18 t/s
b)9.11Ggemma4-coding-Q6_K.ggufsuccess for temperature fails for load averages~ 25 t/s
c)9.11GQwen3.5-9B-Q8_0.ggufsuccess for temperature fails for load averages~ 25 t/s
d)14.84GQwen2.5-Coder-32B-Instruct-Q3_K_M.ggufsuccess for temperature fails slightly for load averages (2/3)~ 1 t/s

Warning:
If you run the below scripts you are responsible. You will see llama-cpp warnings like below so make sure you are safe.
srv llama_server: -----------------
srv llama_server: CORS is set to allow all origins ('*') and no API key is set
srv llama_server: this can be a security risk (cross-origin attacks)
srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
srv llama_server: -----------------
srv llama_server: -----------------
srv llama_server: the following feature(s) are enabled:
srv llama_server: server tools (experimental)
srv llama_server: do not expose the server to untrusted environments
srv llama_server: -----------------


Comments:
- llama-cpp was installed from ports first, that might have helped to build it from source
- for aider the dependencies that caused errors were installed as pkg and then edited out from requirements.txt
- the reason speed drops for models sized > NVRAM is that the CPU/RAM slows everything down when offloading from NVRAM is being done
- there is margin in the memory management in llama startup script to allow kde and chromium to run at the same time as the ai model
- there is a feeling that Vulkan not always is giving memory back after stopping llama-server with ctrl-c

Edit:
The script below has been tweaked and updated.

This script will run models up to ~19GB in size for any crap old hardware with 16GB RAM and 12GB NVRAM which means 32/34B models with Q4_K_M/Q3_K_M. The speed is estimated to around 0.5-1.5 t/s for any model that goes above 12GB. For smaller models speed is estimated to around 20-50 t/s.

Script to start llama and pick a model:

sh:
#!/bin/sh

MODELPATH="$HOME/AI-models"
TOTAL_VRAM=12.0      # Physical VRAM in GB
MAX_LOCKED_RAM=10.0  # FreeBSD limit for mlock in GB

# Path to your working Python 3.12 binary inside your venv
PY_BIN="$HOME/venvs/aider-env/bin/python3"

run_llama() {
    FILE_NAME="$1"
    FILE_SIZE_GB="$2"
    FULL_PATH="$MODELPATH/$FILE_NAME"

    echo "──────────────────────────────────────────────────────"
    echo "📊 Analyzing $FILE_NAME ($FILE_SIZE_GB GB)..."

    # Verify NVIDIA GPU and driver status cleanly
    echo "🎮 Verifying NVIDIA GPU status..."
    if [ ! -c /dev/nvidia0 ]; then
        echo "❌ Error: NVIDIA device node (/dev/nvidia0) is missing!"
        return 1
    fi

    NV_VERSION=$(sysctl -n hw.nvidia.version 2>/dev/null)
    if [ -z "$NV_VERSION" ]; then
        echo "❌ Error: NVIDIA kernel module is not active!"
        return 1
    fi
    echo "✅ GPU Status: NVIDIA Graphics Driver ($NV_VERSION) is active."

    echo "🧠 Reading model built-in metadata live..."

    export TARGET_GGUF_PATH="$FULL_PATH"

    METADATA=$("$PY_BIN" -c '
import sys, struct, os

def read_gguf_metadata(filepath):
    ctx_val, blocks_val = None, None
    try:
        with open(filepath, "rb") as f:
            # Verify Magic Header "GGUF"
            magic = f.read(4)
            if magic != b"GGUF": return None, None
        
            # Read Version (uint32) and V2/V3 fields
            version = struct.unpack("<I", f.read(4))[0]
            if version not in [2, 3]: return None, None
        
            # Read counts
            tensor_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
            kv_count = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
        
            # Types map according to GGUF spec
            # 0=uint8, 1=int8, 2=uint16, 3=int16, 4=uint32, 5=int32, 6=float32, 7=bool, 8=string, 9=array...
            for _ in range(kv_count):
                # Read key string
                key_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                key = f.read(key_len).decode("utf-8", errors="ignore")
            
                # Read value type
                val_type = struct.unpack("<I", f.read(4))[0]
            
                # Helper to skip or read values based on type
                def skip_value(t):
                    if t in [0, 1, 7]: f.seek(1, 1)
                    elif t in [2, 3]: f.seek(2, 1)
                    elif t in [4, 5, 6]: return struct.unpack("<I", f.read(4))[0]
                    elif t == 8: # String
                        s_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        f.seek(s_len, 1)
                    elif t == 9: # Array
                        arr_type = struct.unpack("<I", f.read(4))[0]
                        arr_len = struct.unpack("<Q" if version == 3 else "<I", f.read(8 if version == 3 else 4))[0]
                        for _ in range(arr_len): skip_value(arr_type)
                    elif t in [10, 11, 12, 13]: f.seek(8, 1) # uint64, int64, float64
                    return None

                res = skip_value(val_type)
            
                # Capture our specific targets cleanly
                if "context_length" in key and res:
                    ctx_val = res
                elif "block_count" in key and res:
                    blocks_val = res
                
                if ctx_val and blocks_val:
                    break
    except Exception:
        pass
    return ctx_val, blocks_val

ctx, blocks = read_gguf_metadata(os.environ.get("TARGET_GGUF_PATH", ""))
ctx_str = str(ctx) if ctx else ""
blocks_str = str(blocks) if blocks else ""
print(ctx_str + " " + blocks_str)
')

    TRAINED_CTX=$(echo "$METADATA" | awk '{print $1}')
    ACTUAL_LAYERS=$(echo "$METADATA" | awk '{print $2}')

    if [ -z "$TRAINED_CTX" ] || [ "$TRAINED_CTX" -lt 2048 ]; then
        TRAINED_CTX=32768
        echo "ℹ  Could not read trained context length. Using safe fallback: $TRAINED_CTX"
    else
        echo "🧠 Model is natively trained for a max of: $TRAINED_CTX tokens."
    fi

    if [ -z "$ACTUAL_LAYERS" ] || [ "$ACTUAL_LAYERS" -lt 10 ]; then
        ACTUAL_LAYERS=64
        echo "ℹ  Could not read layer count. Using baseline fallback: $ACTUAL_LAYERS layers."
    else
        echo "🧱 Model has an architectural structure of: $ACTUAL_LAYERS layers."
    fi

#
# MEMORY MANAGEMENT (DYNAMIC VRAM/RAM SPLIT & DYNAMIC KV CACHE)
#
    EVAL_OPTS=$(awk -v vram="$TOTAL_VRAM" -v ram="$MAX_LOCKED_RAM" -v size="$FILE_SIZE_GB" -v max_ctx="$TRAINED_CTX" -v total_layers="$ACTUAL_LAYERS" '
    BEGIN {
        system_vram_overhead = 1.8;
        available_vram = vram - system_vram_overhead;
    
        # SCENARIO 1: Model fits entirely in VRAM -> Use full quality f16 KV Cache
        if (size <= available_vram) {
            cache_type = "f16";
            cache_factor = 0.0065; # Full precision footprint
        
            vram_left_for_cache = available_vram - size;
            calculated_ctx = int((vram_left_for_cache * 1000) / (total_layers * cache_factor));
            ctx = int(calculated_ctx / 1024) * 1024;
        
            if (ctx < 2048) ctx = 2048;
            if (ctx > max_ctx) ctx = max_ctx;
        
            ngl = total_layers + 5;
            ram_fallback = 0;
            b = 512; ub = 256;
            mode = "mlock"; # Safe to lock in RAM since 0 model layers spill over
        }
        # SCENARIO 2: Model requires split-mode -> Switch to q4_0 KV Cache to save VRAM for layers
        else {
            cache_type = "q4_0";
            cache_factor = 0.0017; # Compressed footprint
        
            ctx = 4096;
            if (ctx > max_ctx) ctx = max_ctx;
        
            cache_vram_cost = (ctx / 1000) * total_layers * cache_factor;
            usable_vram_for_layers = available_vram - cache_vram_cost;
        
            if (usable_vram_for_layers < 0) usable_vram_for_layers = 0;
        
            vram_ratio = usable_vram_for_layers / size;
            ngl = int(total_layers * vram_ratio);
        
            if (ngl < 0) ngl = 0;
            if (ngl > (total_layers + 5)) ngl = total_layers + 5;
        
            ram_fallback = size - usable_vram_for_layers;
            b = 512; ub = 256;
            mode = "mmap"; # Use standard mmap to safely spill past FreeBSD mlock limits
        }
    
        if (ram_fallback > ram) {
            print "ERROR: Model requires " ram_fallback " GB system RAM offload, which exceeds your 10 GB limit!" > "/dev/stderr";
            printf "0 ERROR 0 0 0 error\n";
            exit 1;
        }
    
        printf "%d %s %d %d %d %s\n", ctx, cache_type, b, ub, ngl, mode;
    }')

    read -r ctx type b ub ngl mode <<EOF
$EVAL_OPTS
EOF

    if [ "$type" = "ERROR" ] || [ -z "$ctx" ] || [ "$ctx" -eq 0 ]; then
        echo "❌ ERROR: Launch Aborted! Constraints not met."
        return 1
    fi

    # Calculate safe prompt margin
    SAFE_PROMPT_LIMIT=$((ctx - 512))
    if [ "$SAFE_PROMPT_LIMIT" -lt 1024 ]; then
        SAFE_PROMPT_LIMIT=$ctx
    fi

    echo "⚙️  Optimized profile: CTX=$ctx, NGL=$ngl, BATCH=$b, UBATCH=$ub, CACHE=$type, MODE=$mode"
    echo "🔄 Context Shifting: Active (Infinite scrolling chat enabled)"
    echo "💡 Safe Prompt Limit: Do not exceed $SAFE_PROMPT_LIMIT tokens in a SINGLE prompt."
    echo "──────────────────────────────────────────────────────"

    # Start the server with q4_0 KV cache flags and flash attention
    llama-server \
        -t 3 \
        -b "$b" \
        -tb 4 \
        -ub "$ub" \
        -c "$ctx" \
        --context-shift \
        -fa on \
        --load-mode "$mode" \
        -ctk "$type" \
        -ctv "$type" \
        --jinja \
        --host 0.0.0.0 \
        --port 8080 \
        --alias kAI \
        --temp 0.4 \
        --top-p 0.1 \
        --min-p 0.05 \
        --tools all \
        -m "$FULL_PATH"
}

# Check if model directory exists
if [ ! -d "$MODELPATH" ]; then
    echo "❌ Error: Model directory $MODELPATH does not exist."
    exit 1
fi

echo "Pick an AI model to run: "

index=1
chars="123456789abcdefghijklmnopqrstuvwxyz"

MATCHED_MODELS=$(find "$MODELPATH" -maxdepth 1 -name "*.gguf" 2>/dev/null | while read -r path; do
    bytes=$(stat -f %z "$path")
    name=$(basename "$path")
    echo "$bytes $name"
done | sort -n)

old_ifs=$IFS
IFS='
'
for line in $MATCHED_MODELS; do
    IFS=$old_ifs
    bytes=$(echo "$line" | awk '{print $1}')
    name=$(echo "$line" | cut -d' ' -f2-)
    size_gb=$(awk -v b="$bytes" 'BEGIN {printf "%.2f", b / 1024 / 1024 / 1024}')
    selector=$(echo "$chars" | cut -c "$index")
    if [ -z "$selector" ]; then
        break
    fi
    eval "MENU_KEY_$selector=\"$name\""
    eval "MENU_SIZE_$selector=\"$size_gb\""
    printf "%s)   %6sG   %s\n" "$selector" "$size_gb" "$name"
    index=$((index + 1))
    IFS='
'
done
IFS=$old_ifs
echo "0) Exit"

printf "Your choice: "
read choice

if [ "$choice" = "0" ] || [ -z "$choice" ]; then
    echo "Exiting."
    exit 0
fi

eval "SELECTED_MODEL=\$MENU_KEY_$choice"
eval "SELECTED_SIZE=\$MENU_SIZE_$choice"

run_llama "$SELECTED_MODEL" "$SELECTED_SIZE"


/grandpa
FWIW, the lama_run.sh (my name) script fails if the models are stored in a directory tree with depth > 1.
Example:
Code:
tingo@locaal:~ $ find $HOME/.cache//huggingface/hub -name "*.gguf"
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q6_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/mmproj-Qwen_Qwen3.5-27B-bf16.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_M.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_S.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q2_K_L.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--TheDrummer_Orion-26B-A4B-v1.1-GGUF/snapshots/ecc1d958b6b4a83445ab80262965e151e00d97ac/TheDrummer_Orion-26B-A4B-v1.1-IQ3_XS.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--TheDrummer_Orion-26B-A4B-v1.1-GGUF/snapshots/ecc1d958b6b4a83445ab80262965e151e00d97ac/mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/125a02af4987b57c7deb88d7f2ec58a5725c07c0/Qwen3.8-27B-IQ2_XS.gguf
/home/tingo/.cache//huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/125a02af4987b57c7deb88d7f2ec58a5725c07c0/mmproj-Qwen3.8-27B-bf16.gguf
script, modified to use my paths, and I have taken out the 'maxdepth 1' from the find command
when run
Code:
tingo@locaal:~ $ sh bin/run_llama.sh
Pick an AI model to run: 
1)     0.00G   Qwen3.8-27B-IQ2_XS.gguf
2)     0.00G   Qwen_Qwen3.5-27B-Q2_K_L.gguf
3)     0.00G   Qwen_Qwen3.5-27B-Q3_K_L.gguf
4)     0.00G   Qwen_Qwen3.5-27B-Q3_K_M.gguf
5)     0.00G   Qwen_Qwen3.5-27B-Q3_K_S.gguf
6)     0.00G   Qwen_Qwen3.5-27B-Q6_K_L.gguf
7)     0.00G   TheDrummer_Orion-26B-A4B-v1.1-IQ3_XS.gguf
8)     0.00G   mmproj-Qwen3.8-27B-bf16.gguf
9)     0.00G   mmproj-Qwen_Qwen3.5-27B-bf16.gguf
a)     0.00G   mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf
0) Exit
Your choice: 1
──────────────────────────────────────────────────────
📊 Analyzing Qwen3.8-27B-IQ2_XS.gguf (0.00 GB)...
🎮 Verifying NVIDIA GPU status...
✅ GPU Status: NVIDIA Graphics Driver (NVIDIA UNIX x86_64 Kernel Module  595.84  Wed Jun 10 21:13:57 UTC 2026) is active.
🧠 Reading model built-in metadata live...
ℹ  Could not read trained context length. Using safe fallback: 32768
ℹ  Could not read layer count. Using baseline fallback: 64 layers.
⚙️  Optimized profile: CTX=32768, NGL=69, BATCH=512, UBATCH=256, CACHE=f16, MODE=mlock
🔄 Context Shifting: Active (Infinite scrolling chat enabled)
💡 Safe Prompt Limit: Do not exceed 32256 tokens in a SINGLE prompt.
──────────────────────────────────────────────────────
0.00.159.059 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.159.072 I device_info:
0.00.159.143 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
0.00.159.150 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
0.00.159.209 I system_info: n_threads = 3 (n_threads_batch = 4) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
0.00.159.215 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.159.238 I srv          init: running without SSL
0.00.159.270 I srv          init: using 11 threads for HTTP server
0.00.159.388 W srv  llama_server: -----------------
0.00.159.391 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.159.392 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.159.392 W srv  llama_server: -----------------
0.00.159.398 I srv         start: binding port with default address family
0.00.160.545 I srv  llama_server: loading model
0.00.160.552 I srv    load_model: loading model '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.629 I common_init_result: fitting params to device memory ...
0.00.160.631 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.160.664 E gguf_init_from_file: failed to open GGUF file '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf' (No such file or directory)
0.00.160.727 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf
0.00.160.737 E llama_model_load_from_file_impl: failed to load model
0.00.160.771 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.160.779 E gguf_init_from_file: failed to open GGUF file '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf' (No such file or directory)
0.00.160.800 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf
0.00.160.803 E llama_model_load_from_file_impl: failed to load model
0.00.160.804 E common_init_from_params: failed to load model '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.813 E srv    load_model: failed to load model, '/home/tingo/.cache/huggingface/hub/Qwen3.8-27B-IQ2_XS.gguf'
0.00.160.814 I srv    operator(): operator(): cleaning up before exit...
0.00.161.110 E srv  llama_server: exiting due to model loading error
I'm too tired to fix it now. Maybe another night.
 
FWIW, the lama_run.sh (my name) script fails if the models are stored in a directory tree with depth > 1.

tingo@locaal:~ $ sh bin/run_llama.sh
Pick an AI model to run:
1) 0.00G Qwen3.8-27B-IQ2_XS.gguf
2) 0.00G Qwen_Qwen3.5-27B-Q2_K_L.gguf

ℹ Could not read trained context length. Using safe fallback: 32768
ℹ Could not read layer count. Using baseline fallback: 64 layers.

I'm too tired to fix it now. Maybe another night.

Yes, the scripts are particular to my environment and I see you are not getting the file sizes nor the gguf data.

The scripts have evolved to the ones in post #32, #33 where they are maybe just a little bit cleaner. Sorry for any inconvenience.

The later versions are more split up so the gguf reader is now a script of its own. If you want to run with Vulkan you will need to modify the command-to-run and the path to the llama-cpp executable in ai-launcher.sh.

In an attempt to simplify the scripts should be callable with parameters so you can test their output by calling them with actual values for the params.

/grandpa
 
So, while the linuxlator is installed (see #30) it's quite nice to be able to run the latest version of claude.

Talking to myself again.

Here's a technical aspect. It's very difficult to get claude to write to disk if you are low on NVRAM and RAM.

It seems claude wants to use - relatively speaking - large context windows and also there's a general feeling that claude veers towards the anthropic models.

Even with environment variables like --permission-mode bypassPermissions, as large context window as possible, running from a bash terminal and a .git subdirectory it's neigh on impossible to make claude create new files on local disk. It can - sometimes - compile source code and then create executables and it can read files, just not create new ones.

If anyone finds a solution to making claude create new files on systems with low RAM/NVRAM please let me know.

/grandpa
 
Here's a very crude and rough guide for how to pick an AI-model to run locally with purpose to get as much out of the model as possible.

mysterious quant in file namesomething about equivalent bitsnumber of parameters the model's been trained on (B)
BF16 / FP16161-8
Q8_0 / INT881-8
Q6_K / INT66.51-8
Q5_K_M5.51-8
Q5_K_S / Q5_051-14
GPTQ414-32
IQ4_NL4.514-32
Q4_K_M4.514-32
IQ4_XS4.2514-70
Q4_K_S / INT4432-70
IQ3_M3.570+
Q3_K_L 3.570+
IQ3_S / IQ3_XXS370-120
Q3_K_M3.370-120
IQ2 / Q2270+
IQ11.5100+

If you download a model with Q8_0 in the filename then look for a model that has 1B up to 8B in the filename. Or if you download a model with IQ2 then look for 70B and upwards.

/grandpa
 
Another aspect for stubborn old fools like me trying to run AI models on hardware that has parts from 2011 ...

Here' s a neat tool to catch-and-compress the context window before it's being sent to the model.

In effect you can use a context that is up to 3 times larger than the one llama-server has allocated in NVRAM or RAM without llama knowing.

This is the tool on github - sqz

Installation and setup (you will to have rust installed):
cargo install sqz-cli
sqz init --global


Claude should pick up this by automagic but for aider the script run-aider.sh in #33 needs to be changed:
sh:
# ==============================================================================
# 2. APPLY HARD OPTIMISATION AND MEMORY RESTRICTIONS (SQZ ADAPTIVE CHECK)
# ==============================================================================
export AIDER_MAP_TOKENS="0"
export AIDER_MAX_CHAT_HISTORY_TOKENS="8192"
export AIDER_TEMPERATURE="0.1"

# Check if sqz is installed
if [ -x "$HOME/.cargo/bin/sqz" ]; then
    SQZ_BIN="$HOME/.cargo/bin/sqz"
elif command -v sqz >/dev/null 2>&1; then
    SQZ_BIN="sqz"
else
    SQZ_BIN=""
fi
   
if [ -n "$SQZ_BIN" ]; then
    # SQZ found: Expand to virtual context window (3x)
    VIRTUAL_CTX=$(awk -v phys="$SELECTED_CTX" 'BEGIN {print int(phys * 3.0)}')
    export AIDER_MAX_CONTEXT_WINDOW="$VIRTUAL_CTX"
    USE_SQZ="true"
else
    # SQZ not found: Fallback to physical context window
    export AIDER_MAX_CONTEXT_WINDOW="$SELECTED_CTX"
    USE_SQZ="false"
fi
   
# ==============================================================================
# 5. EXECUTE THE NATIVE FREEBSD AIDER ENGINE FLOW WITH POSIX-SAFE SQZ PIPING
# ==============================================================================
echo "🤖 Initialising Aider Client Session..."
if [ "$USE_SQZ" = "true" ]; then
    echo "📊 Context Layer   : SQZ Intelligence Active [ojuschugh1/sqz]"
    echo "📊 Hardware Buffer : ${SELECTED_CTX} tokens | Virtual Window: ${AIDER_MAX_CONTEXT_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    
    # Running with sqz - saving lint output in /tmp and pipe to sqz then read back exit code to aider 
    "$AIDER_BIN_EXEC" \
     --lint-cmd "c: sh -c 'clang -fsyntax-only \"\$@\" > /tmp/aider_lint.log 2>&1; RC=\$?; cat /tmp/aider_lint.log | $SQZ_BIN; exit \$RC'" \
     --test-cmd "sh -c '$TEST_RUNNER > /tmp/aider_test.log 2>&1; RC=\$?; cat /tmp/aider_test.log | $SQZ_BIN; cat /tmp/aider_test.log; exit \$RC'" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
else
    echo "📊 Context Layer   : Native Hardware Direct (SQZ not found)"
    echo "📊 Allocated Buffer: ${AIDER_MAX_WINDOW} tokens"
    echo "────────────────────────────────────────────────────────────────────────"
    
    # No sqz found
    "$AIDER_BIN_EXEC" \
     --lint-cmd "c: clang -fsyntax-only" \
     --test-cmd "$TEST_RUNNER" \
     --git \
     --edit-format diff --no-show-model-warnings --yes-always
fi

exit 0

You can see the output in aider:
.....
cpu_stats.c
Applied edit to cpu_stats.c
Commit 781fcd1 fix: implement cpu temp from sysctl and report system load averages
[sqz] 3/3 tokens (0% reduction)
....


Another way is to run sqz stats:
$ sqz stats

📊 sqz compression stats
──────────────────────────────────────────────────

0 tokens saved
↓ 0.0% average reduction

Compressions 2
Tokens in 7
Tokens out 7
Tokens saved 0
Avg reduction 0.0%

🗄 Cache
──────────────────────────────────────────────────
Entries 1
Size 804 B



/grandpa
 
Talking to myself again.

Here's a technical aspect. It's very difficult to get claude to write to disk if you are low on NVRAM and RAM.
....

Some notes about claude. It autoupdates. That is kind of scary. I can't remember setting it to let itself be updated but I assume it can be turned off somewhere.

Anyway, autoupdate might not be so bad after all since suddenly claude started writing to disk. Of course that could also be the models that I just downloaded and are testing.

If it is model dependent then this model works fine with claude - granite-4.2-3b-Q8_0.gguf and granite-4.2-8b-Q5_K_M.gguf.

/grandpa
 
Another aspect for stubborn old fools like me trying to run AI models on hardware that has parts from 2011 ...

I still have to see a credible explanation about why that's a problem or challenge. What binary operation does it do that can't be replaced by anything else, like 2 systems of 2011 hardware working together?
GPU processing can be virtualized, distributed and simulated. I doubt the central hardware requirement of AI systems. We have flops. What's still not enough? Calculating perspectve in 3d space needs floating point operations too...
 
I still have to see a credible explanation about why that's a problem or challenge. What binary operation does it do that can't be replaced by anything else, like 2 systems of 2011 hardware working together?
GPU processing can be virtualized, distributed and simulated. I doubt the central hardware requirement of AI systems. We have flops. What's still not enough? Calculating perspectve in 3d space needs floating point operations too...

That sentence was more of a personal nature and has nothing to do with it being a problem. For a fool like me it is challenging though since I basically knew very little about how AI works not so long ago. So not a technical aspect. I like to get old things to work.

I still don't know much about how AI works even though I did compile and run a software a couple of years ago that had a model that you fed a series of pictures and the model learned to pick out the pictures that were t-shirts by setting rules.

But I think I understand your question - why does it require so much computational power? Maybe because the data has become vast, maybe because the computations are long and complicated - I don't know.

What I am seeing is that an AI model is a frozen bubble in time. It is not even aware of what time or date it is - unless of course you tell it. Maybe it's like a big bubble of matrix math? If that is the case then John Carmack could probably make it run on a 286 ...

/grandpa
 
...
Here' s a neat tool to catch-and-compress the context window before it's being sent to the model.
...

Small update on this, to make it run with claude it is required to install sqz-mcp.

$ cargo install sqz-mcp


Add .cargo/bin/ to path or edit .claude.json and add:
JSON:
    "sqz": {
      "_sqz_managed": true,
      "command": "~/.cargo/bin/sqz-mcp",
      "args": [
        "--transport",
        "stdio"
      ]
    }

/grandpa
 
That sentence was more of a personal nature and has nothing to do with it being a problem. For a fool like me it is challenging though since I basically knew very little about how AI works not so long ago. So not a technical aspect. I like to get old things to work.

I still don't know much about how AI works even though I did compile and run a software a couple of years ago that had a model that you fed a series of pictures and the model learned to pick out the pictures that were t-shirts by setting rules.

But I think I understand your question - why does it require so much computational power? Maybe because the data has become vast, maybe because the computations are long and complicated - I don't know.

What I am seeing is that an AI model is a frozen bubble in time. It is not even aware of what time or date it is - unless of course you tell it. Maybe it's like a big bubble of matrix math? If that is the case then John Carmack could probably make it run on a 286 ...

/grandpa
There should be an elementary operation that consists of a circuit of transisrtors, like all digital devices. I don't believe it has to be represented by a central module like a GPU.
What's the problem of creating that in only software-logic? Show me the component that nobody can make except AI brokers and must be made of real hardware.
Carmack probably makes a chatgpt equivalent in 486 assembly. Easy... :cool:
 
Back
Top