Running a GPU-accelerated LLM on FreeBSD (2-line howto)

For 32k context size there are these models,

Model ,Total Expected Working VRAM (32k Ctx)

Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.gguf,~12.6 GB
---> slow

Qwen3.6-27B-A3B-Coder-CD-Q6_K.gguf,~12.6 GB

Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf,~16.4 GB

Qwen2.5-32B-Instruct-Q4_K_M

------------------------------------------------------------------------------------------
Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL
---> Fast & Clever
 
Pierre-Marie Baty said:
Extra hint: if you install searx-ng, you can add an extra "web search" tool to your model.
Click to expand...
I have to correct my suboptimal advice here. Forget about searx-ng. I compiled OpenSERP yesterday from its official repo: https://github.com/karust/openserp - the build went straightforward, and it turned out it is a much more capable LLM web search provider than searx-ng. You only need to pass it the path to the chromium binary ( --browser-path /usr/local/bin/chrome), wrap it in a shell script like this one (dependencies: jq for JSON parsing):

sh: 
#!/bin/sh

# read what our caller sends line per line, and only reply when it's correctly formatted MCP JSON
while IFS= read -r REQUEST; do

    REQ_ID="$(echo "${REQUEST}"|jq -c '.id // null')"
    test -n "${REQ_ID}" || continue

    METHOD="$(echo "${REQUEST}"|jq -r '.method // ""')"
    if [ "${METHOD}" = "initialize" ]; then
        jq -cn --argjson id "${REQ_ID}" '{
            jsonrpc: "2.0",
            id: $id,
            result: {
                protocolVersion: "2025-06-18",
                capabilities: { tools: {} },
                serverInfo: { name: "websearch-mcp", version: "1.0.0" }
            }
        }'
        continue
    elif [ "${METHOD}" = "notifications/initialized" ]; then
        continue
    elif [ "${METHOD}" = "ping" ]; then
        jq -cn --argjson id "${REQ_ID}" '{
            jsonrpc: "2.0",
            id: $id,
            result: {
            }
        }'
        continue
    elif [ "${METHOD}" = "tools/list" ]; then
        jq -cn --argjson id "${REQ_ID}" '{
            jsonrpc: "2.0",
            id: $id,
            result: {
                tools: [
                    {
                        name: "web_search",
                        description: "Search the web for current information. Use when the query needs web research.",
                        inputSchema: {
                            type: "object",
                            properties: {
                                query: { 
                                    type: "string",
                                    description: "The search query"
                                },
                                engine: {
                                    type: "string",
                                    enum: [
                                        "google",
                                        "yandex",
                                        "baidu",
                                        "bing",
                                        "duckduckgo",
                                        "ecosia"
                                    ],
                                    description: "Search engine; defaults to google"
                                },
                                limit: {
                                    type: "integer",
                                    minimum: 1,
                                    maximum: 10,
                                    description: "Maximum results; defaults to 5"
                                }
                            },
                            required: [
                                "query"
                            ],
                            additionalProperties: false
                        }
                    }
                ]
            }
        }'
        continue
    elif [ "${METHOD}" = "tools/call" ]; then
        TOOLNAME="$(echo "${REQUEST}"|jq -r '.params.name // ""')"
        if [ "${TOOLNAME}" != "web_search" ]; then
            jq -cn --argjson id "${REQ_ID}" '{
                jsonrpc: "2.0",
                id: $id,
                error: {
                    code: -32602,
                    message: "Unknown tool"
                }
            }'
            continue
        fi
        QUERY="$(echo "${REQUEST}"|jq -r '.params.arguments.query // ""')"
        if [ -z "${QUERY}" ] || [ "${QUERY}" = "null" ]; then
            jq -cn --argjson id "${REQ_ID}" '{
                jsonrpc: "2.0",
                id: $id,
                result: {
                    content: [
                        {
                            type: "text",
                            text: "Error: query must not be empty"
                        }
                    ],
                    isError: true
                }
            }'
            continue
        fi
        ENGINE="$(echo "${REQUEST}"|jq -r '.params.arguments.engine // "google"')"
        if ! echo "${ENGINE}"|grep -q -e '^google$' -e '^yandex$' -e '^baidu$' -e '^bing$' -e '^duckduckgo$' -e '^ecosia$'; then
            jq -cn --argjson id "${REQ_ID}" '{
                jsonrpc: "2.0",
                id: $id,
                result: {
                    content: [
                        {
                            type: "text",
                            text: "Error: unsupported search engine"
                        }
                    ],
                    isError: true
                }
            }'
            continue
        fi
        LIMIT="$(echo "${REQUEST}"|jq -r '.params.arguments.limit // 5')"
        if ! echo "${LIMIT}"|grep -q -e '^[1-9]$' -e '^10$'; then
            jq -cn --argjson id "${REQ_ID}" '{
                jsonrpc: "2.0",
                id: $id,
                result: {
                    content: [
                        {
                            type: "text",
                            text: "Error: limit must be an integer from 1 to 10"
                        }
                    ],
                    isError: true
                }
            }'
            continue
        fi
        RESULT="$(openserp --browser-path /usr/local/bin/chrome --limit "${LIMIT}" --format json search "${ENGINE}" "${QUERY}" 2>/dev/null)"
        if [ -z "${RESULT}" ]; then
            jq -cn --argjson id "${REQ_ID}" '{
                jsonrpc: "2.0",
                id: $id,
                result: {
                    content: [
                        {
                            type: "text",
                            text: "OpenSERP search failed. Check the CLI installation and search engine configuration."
                        }
                    ],
                    isError: true
                }
            }'
            continue
        fi
        jq -cn --argjson id "${REQ_ID}" --arg result "${RESULT}" '{
            jsonrpc: "2.0",
            id: $id,
            result: {
                content: [
                    {
                        type: "text",
                        text: $result
                    }
                ]
            }
        }'
        continue
    else
        jq -cn --argjson id "${REQ_ID}" '{
            jsonrpc: "2.0",
            id: $id,
            error: {
                code: -32601,
                message: "Method not found"
            }
        }'
        continue
    fi
done
Save this one as ~/bin/websearch-mcp.sh, add a JSON file that will instruct llama.cpp about it:
JSON: 
{
  "mcpServers": {
    "openserp": {
      "command": "/home/youraccountname/bin/websearch-mcp.sh",
      "args": []
    }
  }
}
Run llama.cpp with --jinja --tools all --mcp-servers-config ~/llamacpp-mcp.json and there, you've got a web-enabled local LLM :)
 
Alain De Vos said:
Interesting concept.
Click to expand...
Thank you. While using it I noticed that the model used bare curl calls piped through ingenuous sed/awk filters to extract webpage contents of the URLs reported to it by the MCP server. To make its job a little easier, I patched openserp to add a "fetch" command to the CLI, and extended this minimal Bourne shell MCP server with a "fetch_page" command. If someone's interested in the patches I can post them here (or start a new thread, as you prefer.)
 
fyi,

Code: 
dotnet run
=== Available Models ===
1. DeepSeek-R1-Distill-Qwen-1.5B-Q8_0  
2. Qwen2.5-Coder-7B
3. Qwen2.5-Coder-7B-Instruct-Q6_K      
4. Empty-Model-Mock
5. Qwen2.5-Coder-7B-Instruct-Q8_0      
6. Qwen3.6-27B-Fable-Fus-MTP-Q4_K_M    !!
7. Qwen3.6-27B-UD-Q5_K_XL              
8. Qwen3.8-27B-Coder-EffThink-Q6_K-MTP !!
9. Qwen3.6-27B-A3B-Coder-CD-Q6_K       
10. Qwen2.5-Coder-14B-Instruct-Q4_K_M   
11. Qwen2.5-Coder-14B-Instruct-Q5_K_M   
12. SLOW----------------------------------
13. Qwen3.6-35B-A3B-UD-IQ4_XS           
14. Qwen2.5-32B-Instruct-Q4_K_M         
15. Qwen3.6-35B-A3B-MTP-UD-Q4_K_M       
16. Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL      
17. Qwen3.6-35B-A3B-UD-Q4_K_XL          
18. Qwen3.5-35B-A3B-bf16-Split1         
19. NEW-----------------------------------
20. Qwen3.6-27B-UD-Q5_K_XL(bis)         
21. Qwen3-Coder-Next-Q5_K_M             
Select a model (1-21):

Model 21 is a supermodel :).
Model 16 is for everyday use.

Llama-launcher code,
https://gitlab.com/alaindevos/scalatut/-/blob/master/llama/launch_sharp/Program.fs?ref_type=heads
 
This gives good results,
Code: 
Selected Model: Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL        
[Llama Server Debug] Spawning command:
llama-server \
--batch-size 512 --cache-ram 4192 --chat-template-kwargs {"preserve_thinking":true} \
--cpu-moe --ctx-size 8192 --fit off --flash-attn on --host 127.0.0.1 --jinja --kv-unified \
--load-mode mmap+mlock --min-p 0.05 \
--model /mnt/MYWINZFS/data/HUGGING/GUFF/C/Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \
--n-cpu-moe 34 --n-gpu-layers 40 --no-reasoning-preserve --no-warmup --parallel 2 --port 8123 \
--presence-penalty 0.0 --reasoning on --reasoning-budget -1 --repeat-penalty 1.0 --spec-draft-n-max 2 \
--spec-type draft-mtp --split-mode layer --sse-ping-interval 15 --temp 0.05 --threads 9 --threads-batch 9 \
--timeout 7200 --top-k 1 --top-p 0.95 --ubatch-size 512 --verbose

And
Code: 
Selected Model: Qwen3-Coder-Next-Q5_K_M              
[Llama Server Debug] Spawning command:
llama-server \
--batch-size 512 --cache-ram 4192 --chat-template-kwargs {"preserve_thinking":true} \
--cpu-moe --ctx-size 8192 --fit off --flash-attn on --host 127.0.0.1 --jinja --kv-unified \
--load-mode mmap+mlock --min-p 0.05 \
--model /mnt/MYWINZFS/data/HUGGING/GUFF/psy/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-Q5_K_M/Qwen3-Coder-Next-Q5_K_M_merged.gguf \
--n-cpu-moe 36 --n-gpu-layers 36 --no-reasoning-preserve --no-warmup --parallel 2 --port 8123 \
--presence-penalty 0.0 --reasoning on --reasoning-budget -1 --repeat-penalty 1.0 --split-mode layer \
--sse-ping-interval 15 --temp 0.95 --threads 9 --threads-batch 9 --timeout 7200 --top-k 40 --top-p 0.95 \
--ubatch-size 512 --verbose
This last one gives me 1.3t/s . But you can ask it really difficult questions ...
 
At the other end of the spectrum - the old and crap hardware dept - it is noticable that the chat template has an astonishing effect on what a (small) model can do or not. It can be the difference between being able to use tools or not or hallucinating or not.

The current version of llama server I run with CUDA in the linuxlator says this about chat-templates:

list of built-in templates:
bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml,
command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe,
exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite,
granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe,
hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos,
llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1,
mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch,
openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss,
smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr
(env: LLAMA_ARG_CHAT_TEMPLATE_FILE)

Then of course you can write your own and use instead of the built-in ones.

/grandpa
 
Tuning for good "--n-gpu-layers X --n-cpu-moe Y" is a dark art.
I developped some unscientific heuristics.

Let Z be total layers of the model.
- Start with X=Z/2 and Y=Z

If this works :
-> look at nvidia-smi
-> Then let X_new=X_old* Memory_Card / Usage_card
If it does not work :
-> let X_new=X_old/2

..... etc ...

Once you have max layers in gpu it's time to tune --cpu-moe
start with Y=Z
Then do a binary search towards Yoptimal, that is the lowest Y_value ,
Y_new=Z-sqrt( (Z-Y_oom)*( Z-Y_good))

Another heuristic is : --n-gpu-layers + --cpu-moe = Total_layers_of_card , but that one is very unscientific :)
 
You caught my attention. I have a calculation for offloading to cpu in my calc- scripts but I don't differentiate for moe/weight layers which seems clever.

I will need to go back and modify my scripts to consider moe and weights.

It looks like these gguf parameters can give the moe size - embedding_length + feed_forward_length + expert_count?

/grandpa
 
Modified the scripts so they fill the moe_experts in the vram that's left from vram - attention_parts - kv_cache and let the rest spill over into ram. Can't say that I see a noticable performance increase though. It's almost worse than just doing blind offloading with ngl only?

Was this a rabbit hole?

I based attention_parts on embedding_length from the gguf data and used a 17Gb large qwen3-coder-30B-a3b-instruct as test model. Of course my system is heavily unbalanced making anything in CPU horribly slow so that might hide any performance gain.

/grandpa
 
Czech startup bottlecap AI (lol, czech humor) founded by the LLM pioneer (like the real one Phd, without whom it would not happen) have some interesting thinking optimized google and qwen models. The best ones are not free, tho.
Gonna test them this week. You might too.

Screenshot_2026-10-29_12-20-19.png
 
cracauer@ said:
Code: 
pkg install llama-cpp
llama-server \
        --host `hostname` \
        --port 8080 \
        --ctx-size $((64 * 1024)) \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L

This is assuming you already have a working graphics card driver with Vulkan. You can test with `vulkaninfo --summary` from pkg vulkan-tools. Working Vulkan is in the binary NVidia drivers and the AMD drivers (including CPU-integrated GPU). Dunno about Intel GPUs.

This will install the llama.cpp LLM runner, which is an intermediate difficulty server. It provides a web server on port 8080 (or as given on the commandline). The FreeBSD port and pkg is compiled with Vulkan by default. The second line downloads the model I like for coding from huggingface.com and starts the server on it. You can use it in a web browser like a commercial LLM on the web. It provides an API on the webserver at the same time, so you can also use it from e.g. Gptel, an Emacs interface to LLMs. I set the temperature pretty low (less creativity and fantasizing) because that is what the model makers recommend for coding. For chit-chat you want to set it to 1.0 (the maximum).

This is a 28 GB model which fits my 32 GB NVidia graphics card. There are hundreds of models of all sizes available on huggingface, let me know what your hardware is and what you want to do for a recommendation.

FreeBSD/Vulkan runs this about 4% slower than Linux/Vulkan on the same hardware, which is 8% slower than Linux/CUDA. At this time I have not succeeded in running CUDA through Linuxulator.
Click to expand...
what about intel drivers? like the intel atom
 
You must log in or register to reply here.
Back
Top