Running a GPU-accelerated LLM on FreeBSD (2-line howto)

cracauer@

cracauer@

Developer
Code: 
pkg install llama-cpp
llama-server \
        --host `hostname` \
        --port 8080 \
        --ctx-size $((64 * 1024)) \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L

This is assuming you already have a working graphics card driver with Vulkan. You can test with `vulkaninfo --summary` from pkg vulkan-tools. Working Vulkan is in the binary NVidia drivers and the AMD drivers (including CPU-integrated GPU). Dunno about Intel GPUs.

This will install the llama.cpp LLM runner, which is an intermediate difficulty server. It provides a web server on port 8080 (or as given on the commandline). The FreeBSD port and pkg is compiled with Vulkan by default. The second line downloads the model I like for coding from huggingface.com and starts the server on it. You can use it in a web browser like a commercial LLM on the web. It provides an API on the webserver at the same time, so you can also use it from e.g. Gptel, an Emacs interface to LLMs. I set the temperature pretty low (less creativity and fantasizing) because that is what the model makers recommend for coding. For chit-chat you want to set it to 1.0 (the maximum).

This is a 28 GB model which fits my 32 GB NVidia graphics card. There are hundreds of models of all sizes available on huggingface, let me know what your hardware is and what you want to do for a recommendation.

FreeBSD/Vulkan runs this about 4% slower than Linux/Vulkan on the same hardware, which is 8% slower than Linux/CUDA. At this time I have not succeeded in running CUDA through Linuxulator.
 
Here is how to run the uncensored model that loveydovey was talking about in
loveydovey

Thread 'Completely uncensored Qwen3.6 released for FreeBSD'

The only limitation is your fantasy now.

The model does not refuse - generates anything you ask it for (in testing it did not refuse to answer any of the 465 dangerous queries).
Can create anything you want - from porn novels to instructions on how to create a nuclear bomb - at your own risk.
Works on almost any hardware - available versions from 11.7 to 30.5 GB.
Works through ollama and other software.
Completely local.

Post your most intriguing answers here.

PS. I think we need a separate forum topic for AI. Don't you all think it's way overdue?

This time with a direct download, not using llama's internal cache. You can store the model anywhere you want.

Code: 
fetch https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive/resolve/main/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf
llama-server \
        --host `hostname` \
        --port 8080 \
        --temp 0.6 \
        --ctx-size $((256 * 1024)) \
        --model ~/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf

On the surface this is a very similar model of comparable on-disk size to the one in the first post, with bumping the qwen version from 3.5 to 3.6. But it is actually quite different. This model is a mixture of experts model with a higher number of base parameters but a lower number of active parameters. In the first post we had a dense model with all parameters active. Of course it has also been uncensored as the name implies.

The MoE model is less stable than the dense one, I get pretty noticeable variations in output for the same input. Overall I find the dense model to be more useful to me.

It is also more than 3x faster. I use most of that speed advantage to bump up the context size to 256k (which is very high for a local model).
 
Heads up: version 8895 of llama.cpp has been committed to ports.

This changes the location where llama.cpp stores models automatically downloaded from huggingface (as in post #1 here).
Old cache: ~/.cache/llama.cpp/
New cache: ~/.cache/huggingface/hub

It will automatically move already downloaded models to the new location. If you have different filesystems under there you might be in for a surprise.

In my case the homedir is on NFS and there is a symlink for the old cache to a local NVMe SSD. So it moved things to NFS and afterwards I made to move them back via a new symlink to the SSD. Took 3 hours I'm never getting back.
 
Quick question do you have to run llama-server as root?
When I try as my user, I get an error message
Code: 
tingo@locaal:~ $ llama-server \
>         --host `hostname` \
>         --port 8080 \
>         --ctx-size $((64 * 1024)) \
>         --temp 0.6 \
>         --top-p 0.95 \
>         --top-k 20 \
>         -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L
Downloading mmproj-Qwen_Qwen3.5-27B-bf16.gguf ────────────────────── 100%
Downloading Qwen_Qwen3.5-27B-Q6_K_L.gguf ─────────────────────────── 100%
12.34.224.670 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
12.34.224.674 I device_info:
12.34.225.223 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
12.34.225.244 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
12.34.225.307 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
12.34.225.315 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
12.34.226.729 I srv          init: running without SSL
12.34.226.755 I srv          init: using 11 threads for HTTP server
12.34.227.973 I srv         start: binding port with default address family
12.34.240.500 E srv         start: couldn't bind HTTP server socket, hostname: locaal, port: 8080
12.34.240.503 I srv    operator(): operator(): cleaning up before exit...
12.34.241.463 E srv  llama_server: exiting due to HTTP server error
this is a high port, so it shouldn't be because of that.
 
To be honest looking at the reasoning that model was doing when I asked the "top censored question" - my jaw almost dropped. Because of my HW limitations I used Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-IQ2_M.gguf.
 
Tip: add --jinja and --tools all to recent llama.cpp releases to have the model use tools that enable it to access files on your host machine (beware: don't let this kid rummage around unsupervised!)

FYI, here's how I launch Qwen 3.8 these days:
llama serve \
--model Downloads/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \
--spec-draft-model Downloads/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-FastMTP-32K.gguf \
--spec-draft-ngl all \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-p-min 0 \
--mmproj Downloads/mmproj-Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-BF16.gguf --image-min-tokens 1024 \
--host 127.0.0.1 --port 65156 \
--ctx-size 204800 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 512 \
--n-gpu-layers all \
--split-mode none \
--flash-attn on \
--temp 1.0 \
--top-k 20 \
--top-p 0.95 \
--min-p 0 \
--presence-penalty 0 \
--repeat-penalty 1.0 \
--jinja \
--reasoning on \
--reasoning-effort xhigh \
--reasoning-preserve \
--reasoning-format deepseek \
--tools all
Click to expand...
Note: the companion .mmproj file is the multimodal projections weights file that enables the model to interpret image data (you can upload pictures in the llama.cpp web UI for the model to reason about).

Note 2: the most recent llama.cpp versions (about to arrive in the ports tree) no longer have llama-serve, everything is condensed in a single binary called llama.

Extra hint: if you install searx-ng, you can add an extra "web search" tool to your model.

*edit* I forgot that for this very specific version of Qwen3.8, a little patch is needed (provided on the HauhauCS HuggingFace page) to apply to llama.cpp to let it make use of the HauhauCS quantization format. Failing to apply this patch may degrade the performance of their models in llama.cpp!
 
Yes, that is correct. You can run anything that fits into 6GB. You will however need fit the model itself, a context window size and some 4 different kind of caches and room for overhead in those 6GB.

And you are not restricted to 6GB. If your setup has RAM models with the above mentioned extra stuff can "spill over" into RAM by utilizing as much as possible of your GPU first and then place the rest in normal RAM. There is a serious drawback to doing this and that is that normal RAM is slower than GPU RAM so while a model that only uses your 6GB runs at acceptable speed a model that "spills over" into RAM will be much much much slower.

/grandpa
 
So... what hardware spec PC do I need to run an LLM this way. You mentioned a 32GB nvidia graphics card.. which type of nvidia GPU do I need? What host processor? How much memory in the host machine? How much storage? Is it just a PC with an nvidia graphics card, or do I need any other parts as well?
 
blackbird9 said:
So... what hardware spec PC do I need to run an LLM this way. You mentioned a 32GB nvidia graphics card.. which type of nvidia GPU do I need? What host processor? How much memory in the host machine? How much storage? Is it just a PC with an nvidia graphics card, or do I need any other parts as well?
Click to expand...
I wonder what happens to the performance when CUDA is secretly a program that does the same binary operations. Something is very obscure about this.
 
blackbird9 said:
So... what hardware spec PC do I need to run an LLM this way. You mentioned a 32GB nvidia graphics card.. which type of nvidia GPU do I need? What host processor? How much memory in the host machine? How much storage? Is it just a PC with an nvidia graphics card, or do I need any other parts as well?
Click to expand...
Just assume that it's a top-end RTX model, like an RTX 3090 or above. And, IIRC, OP has mentioned 128 GB of DDR4 RAM elsewhere on the Forums. Dunno if those are part of the same machine or not.

Sometimes, a low-end GPU is busy enough running just the graphical desktop and playing a lo-res movie.

For running ollama, assume a high-end spec sheet.
 
blackbird9 said:
So... what hardware spec PC do I need to run an LLM this way. You mentioned a 32GB nvidia graphics card.. which type of nvidia GPU do I need? What host processor? How much memory in the host machine? How much storage? Is it just a PC with an nvidia graphics card, or do I need any other parts as well?
Click to expand...

32 GB VRAM is already very tight, although the NVidia 5090 I have is at least fast within those 32 GB. Many interesting models are a tight squeeze and can't have enough context window.

The alternative is an AMD strix halo system which has up to 128 GB RAM, which is shared by the GPU. It is much slower than the 5090, but you can run much larger models. I do not know how well this is supported in FreeBSD.

For chatting with a model you want as a minimum 5 tokens/sec, and that is really slow, just bearable. From 50 or so tokens/sec most people won't notice a further speedup. But agentic work will of course speed up.

What is your budget? For playing with a smaller LLM just for learning, not for "production", a NVidia 2080ti with its 11 GB VRAM is not bad, and it costs about $250.

Keep in mind that buying the hardware is not the end of the cost. Power consumption can be substantial when you have agents noodling around much of the time. And it gets worse if you also need to pay for A/C to get rid of the heat.

Other hardware is not critical as long as you can store the LLM and have more RAM than VRAM.
 
Having 12GB of VRAM and 32GB of RAM, this is my setup,
x@myfreebsd $ cat mylama_deepseek_q5_scala
sh: 
#!/bin/sh
killall llama-server ; sleep 1
killall -9 llama-server ; echo "sleeping 6" ; sleep 6 ; echo "done sleeping"
OLLAMA_BLOBS="/OLLAMA/blobs"
MANIFEST_DIR="/OLLAMA/manifests/registry.ollama.ai/library"
MODEL_NAME="deepseek-coder-v2"
MODEL_TAG="16b-lite-instruct-q5_K_M"
MANIFEST_FILE="$MANIFEST_DIR/$MODEL_NAME/$MODEL_TAG"
echo "Target Model:   $MODEL_NAME:$MODEL_TAG"
echo "Manifest File:  $MANIFEST_FILE"
echo "Blobs Directory: $OLLAMA_BLOBS"
if [ ! -f "$MANIFEST_FILE" ]; then
    echo "❌ ERROR: Manifest file does not exist at $MANIFEST_FILE"
    exit 1
else
    echo "✅ SUCCESS: Manifest file found."
fi
head -c 1000 "$MANIFEST_FILE"
MODEL_BLOCK=$(awk -v RS='}' '/"mediaType":"application\/vnd\.ollama\.image\.model"/{print}' "$MANIFEST_FILE")
echo "1. Isolated Model Block Content:"
echo "   $MODEL_BLOCK"
RAW_DIGEST=$(echo "$MODEL_BLOCK" | grep -o '"digest":"sha256:[^"]*"' | head -n 1 | cut -d'"' -f4)
echo "2. Extracted Raw Digest Signature: $RAW_DIGEST"
BLOB_NAME=$(echo "$RAW_DIGEST" | sed 's/:/-/')
echo "3. Transformed Filename Format:     $BLOB_NAME"
if [ -z "$BLOB_NAME" ] || [ "$BLOB_NAME" = "-" ]; then
    echo "❌ ERROR: Extraction failed! The output blob name is empty or invalid."
    exit 1
fi
MODEL_PATH="$OLLAMA_BLOBS/$BLOB_NAME"
echo "4. Absolute Final Model Path:      $MODEL_PATH"
if [ ! -f "$MODEL_PATH" ]; then
    echo "❌ ERROR: Target GGUF binary does not exist at $MODEL_PATH"
    exit 1
else
    echo "✅ SUCCESS: Valid binary target discovered. File size: $(du -h "$MODEL_PATH" | cut -f1)"
fi
echo "🚀 Launching Llama Server Instance"
CTX_SIZE=8192             # ->49152
CACHE_RAM=2048       #  ->16348
UBATCH_SIZE=512       #   ->1024
llama-server \
        --ctx-size    $CTX_SIZE \
        --cache-ram   $CACHE_RAM \
        --ubatch-size $UBATCH_SIZE \
        --threads-batch 3 \
        --threads 3 \
        --host "127.0.0.1" \
        --port 8123 \
        --temp 0.3 \
        --model "$MODEL_PATH" \
        --no-warmup \
        --parallel 1 &
sleep 5
firefox http://127.0.0.1:8123

Currently downloading
snapshot_download(
repo_id="unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
allow_patterns=["*Q5_K_M*.gguf", "*UD-Q5_K_M*"],
Takes some time ...

Running a a local gguf;
cat myserver_qwen
sh: 
#!/bin/sh
killall llama-server ; sleep 1
killall -9 llama-server ; echo "sleeping 6" ; sleep 6 ; echo "done sleeping"
MODEL_PATH="Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf"
echo "Target Model Path: $MODEL_PATH"
if [ ! -e "$MODEL_PATH" ]; then
    echo "❌ ERROR: Target GGUF binary does not exist at $MODEL_PATH"
    exit 1
else
    echo "✅ SUCCESS: Valid binary target discovered. File size: $(du -h "$MODEL_PATH" | cut -f1)"
fi
echo "🚀 Launching Llama Server Instance"
CTX_SIZE=8192             # ->49152
CACHE_RAM=2048            # ->16348
UBATCH_SIZE=512           # ->1024
llama-server \
        --ctx-size    $CTX_SIZE \
        --cache-ram   $CACHE_RAM \
        --ubatch-size $UBATCH_SIZE \
        --threads-batch 3 \
        --threads 3 \
        --host "127.0.0.1" \
        --port 8123 \
        --temp 0.3 \
        --model "$MODEL_PATH" \
        --no-warmup \
        --parallel 1 &

sleep 5
firefox http://127.0.0.1:8123

Other models,
Code: 
NAME                                        SIZE
deepseek-coder-v2:16b-lite-instruct-q5_K_M  11GB
deepseek-coder:6.7b                         3.8GB
deepseek-r1:1.5b                            1.1GB
dolphincoder:15b                            9.1GB
nuextract:3.8b                              2.2GB
ollama.com/cniongolo/biomistral:latest      4.4GB
phi4:14b                                    9.1GB
qwen2.5-coder:14b-instruct-q5_K_M           10GB
qwen2.5-coder:3b                            1.9GB
qwen2.5-coder:3b-instruct-q5_K_M            2.2GB
qwen2.5-coder:7b-instruct                   4.7GB
qwen2.5-coder:7b-instruct-q5_K_M            5.4GB
qwen3-coder:30b                             18GB
sqlcoder:7b                                 4.1GB
yi-coder:1.5b                               866MB
 
Last edited:
This is gold. Thanks.

Ramblings - to argue with myself and to possibly get some insight - follows:
[uses of '-' I learned form my favorite author R.A. Heinlein and find them useful as English seems sparse with use of ',' . i.e. this was hand typed by meatbag]
I have 16GB vRAM nVIDIA 5060Ti and 32GB RAM on Debian. Currently on OLLAMA but soon switching to llama.cpp.
Not using it often. When I played with it (through Home Assistant Voice mostly) it was the first time I had to pay extra for already high electric bill.
I was thinking of spending cca 4,500 USD on Chinese Strix Halo (AMD with 128GB of shared RAM) to fit larger models and save on electricity but
a) for Home Assistant it might be too slow - no biggie as it is just a toy
b) for serious help, frontier online models are still necessary and 4K can right now pay for decades of use - even if they hike prices 100%, it still makes sense, minus electric bill.
c) I am a little worried that 4K machine will crap out on me just after warranty runs out.

But yeah, for privacy either personal or business this might be necessary expenditure or if $ is not an issue.

This guy has some good pointers:

Interesting usage of local AI, tho I would use Hermes for this instead of OpenWebUI :
 
cracauer@ said:
Yes. Linux Vulkan is 5% slower than Linux CUDA. FreeBSD Vulkan is 5% slower than Linux Vulkan. That's using NVidia.
Click to expand...
I've been searching far and wide for a working xorg.conf [ or set of them ] for such a card as well as the precise directory to put it in and the nvidia driver to load for it. :confused:posting of one would be appreciated. thanks in advance.
 
jb_fvwm2 said:
I've been searching far and wide for a working xorg.conf [ or set of them ] for such a card as well as the precise directory to put it in and the nvidia driver to load for it. :confused:posting of one would be appreciated. thanks in advance.
Click to expand...
I don't use a Xorg.conf.

Just starting X does it.
 
fyi, hugging files downloader script

Python: 
#!/usr/bin/env python3.12

# I was created using A.I.
# I was tuned for 12GB Nvram without spill-over

import os
import fnmatch

# 1. Force the ultra-fast Rust downloader (bypasses tqdm progress bugs)
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

# 2. Quiet the verbose huggingface/tqdm warnings if desired
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1"

from huggingface_hub import snapshot_download

# Defined queue configuration across all your system VRAM profiles (8, 10, 12, 14, 16)
DOWNLOAD_QUEUE = [
    # ==================== 8GB VRAM TIER ====================
    {
        "name": "Qwen2.5-Coder-7B-Instruct (Q5_K_M) [~5.4 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen2.5-Coder-7B-Instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 3,
    },
    {
        "name": "DeepSeek-R1-Distill-Qwen-1.5B (Q8_0) [~1.1 GB Ollama Equiv]",
        "repo_id": "unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF",
        "allow_patterns": ["*Q8_0*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 3,
    },
    {
        "name": "DeepSeek-Coder-6.7B-Instruct (Q5_K_M) [~3.8 GB Ollama Equiv]",
        "repo_id": "bartowski/deepseek-coder-6.7b-instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 3,
    },
    {
        "name": "Qwen2.5-Coder-3B-Instruct (Q5_K_M) [~2.2 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen2.5-Coder-3B-Instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 3,
    },
    {
        "name": "Qwen2.5-Coder-3B-Instruct (Q4_K_M) [~1.9 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen2.5-Coder-3B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 3,
    },
    {
        "name": "Qwen3-8B-Instruct (Q4_K_M) [~5.2 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen3-8B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 4,
    },
    {
        "name": "Qwen3-4B-Instruct (Q4_K_M) [~2.5 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen3-4B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/8",
        "max_workers": 2,
    },

    # ==================== 10GB VRAM TIER ====================
    {
        "name": "Qwen2.5-Coder-14B-Instruct (Q4_K_M) [~9.0 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen2.5-Coder-14B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/10",
        "max_workers": 4,
    },
    {
        "name": "Qwen2.5-Coder-7B-Instruct (Q8_0) [~7.4 GB File]",
        "repo_id": "unsloth/Qwen2.5-Coder-7B-Instruct-GGUF",
        "allow_patterns": ["*Q8_0*.gguf"],
        "local_dir": "/home/x/hugging/10",
        "max_workers": 3,
    },
    {
        "name": "Qwen3-14B-Instruct (Q4_K_M) [~9.3 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen3-14B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/10",
        "max_workers": 4,
    },
    {
        "name": "DeepSeek-Coder-V2-Lite-16B (IQ4_XS) [~8.6 GB File]",
        "repo_id": "bartowski/DeepSeek-Coder-V2-Instruct-GGUF",
        "allow_patterns": ["*IQ4_XS*.gguf"],
        "local_dir": "/home/x/hugging/10",
        "max_workers": 4,
    },

    # ==================== 12GB VRAM TIER (YOUR 12GB SWEET SPOT) ====================
    {
        "name": "Qwen2.5-Coder-14B-Instruct (Q4_K_M) [~9.0 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen2.5-Coder-14B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/12",
        "max_workers": 4,
    },
    {
        "name": "Qwen3-14B-Instruct (Q4_K_M) [~9.3 GB Ollama Equiv]",
        "repo_id": "unsloth/Qwen3-14B-Instruct-GGUF",
        "allow_patterns": ["*Q4_K_M*.gguf"],
        "local_dir": "/home/x/hugging/12",
        "max_workers": 4,
    },
    {
        "name": "DeepSeek-Coder-V2-Lite-16B (Q5_K_M) [~11.0 GB Ollama Equiv]",
        "repo_id": "bartowski/DeepSeek-Coder-V2-Instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/12",
        "max_workers": 4,
    },
    {
        "name": "DeepSeek-Coder-6.7B-Instruct (Q8_0) [~7.2 GB File]",
        "repo_id": "bartowski/deepseek-coder-6.7b-instruct-GGUF",
        "allow_patterns": ["*Q8_0*.gguf"],
        "local_dir": "/home/x/hugging/12",
        "max_workers": 4,
    },
    {
        "name": "Qwen3.6-35B-A3B (UD-IQ3_S)",
        "repo_id": "unsloth/Qwen3.6-35B-A3B-GGUF",
        "allow_patterns": ["*UD-IQ3_S*.gguf"],
        "local_dir": "/home/x/hugging/12",
        "max_workers": 8,
    },

    # ==================== 14GB VRAM TIER ====================
    {
        "name": "Qwen3.6-35B-A3B (UD-IQ4_XS)",
        "repo_id": "unsloth/Qwen3.6-35B-A3B-GGUF",
        "allow_patterns": ["*UD-IQ4_XS*.gguf"],
        "local_dir": "/home/x/hugging/14",
        "max_workers": 8,
    },
    {
        "name": "Qwen2.5-Coder-14B-Instruct (Q5_K_M) [~10.1 GB File]",
        "repo_id": "unsloth/Qwen2.5-Coder-14B-Instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/14",
        "max_workers": 6,
    },
    {
        "name": "DeepSeek-Coder-V2-Lite-16B (Q5_K_M) [~11.0 GB Ollama Equiv]",
        "repo_id": "bartowski/DeepSeek-Coder-V2-Instruct-GGUF",
        "allow_patterns": ["*Q5_K_M*.gguf"],
        "local_dir": "/home/x/hugging/14",
        "max_workers": 5,
    },

    # ==================== 16GB VRAM TIER ====================
    {
        "name": "Qwen2.5-Coder-14B-Instruct (Q8_0) [~15.0 GB File]",
        "repo_id": "unsloth/Qwen2.5-Coder-14B-Instruct-GGUF",
        "allow_patterns": ["*Q8_0*.gguf"],
        "local_dir": "/home/x/hugging/16",
        "max_workers": 5,
    },
    {
        "name": "DeepSeek-Coder-V2-Lite-16B (Q8_0) [~11.9 GB File]",
        "repo_id": "bartowski/DeepSeek-Coder-V2-Instruct-GGUF",
        "allow_patterns": ["*Q8_0*.gguf"],
        "local_dir": "/home/x/hugging/16",
        "max_workers": 6,
    },
    {
        "name": "Qwen3.8-27B-UD (IQ3_S)",
        "repo_id": "unsloth/Qwen3.8-27B-GGUF",
        "allow_patterns": ["*IQ3_S*.gguf"],
        "local_dir": "/home/x/hugging/16",
        "max_workers": 6,
    },
]

def main():
    total_models = len(DOWNLOAD_QUEUE)
    print(f"🚀 Initializing complete model sync pipeline: {total_models} items targets.")
    print("="*70)

    for index, target in enumerate(DOWNLOAD_QUEUE, start=1):
        print(f"\n[{index}/{total_models}] Syncing -> {target['name']}")
        print(f"📂 Location: {target['local_dir']}")
       
        os.makedirs(target['local_dir'], exist_ok=True)
       
        already_exists = False
        try:
            flat_files = os.listdir(target['local_dir'])
            for filename in flat_files:
                if os.path.isfile(os.path.join(target['local_dir'], filename)):
                    for pattern in target['allow_patterns']:
                        if fnmatch.fnmatch(filename, pattern):
                            already_exists = True
                            break
                if already_exists:
                    break
        except Exception:
            pass

        if already_exists:
            print(f"⏭️ File matching {target['allow_patterns']} found at top level. Skipping download.")
            continue
       
        try:
            snapshot_download(
                repo_id=target["repo_id"],
                allow_patterns=target["allow_patterns"],
                local_dir=target["local_dir"],
                max_workers=target["max_workers"]
            )
            print(f"✨ Successfully synced {target['name']}")
        except Exception as e:
            print(f"❌ Error downloading {target['name']}: {e}")

    print("\n🎉 Global system profile synchronization complete!")

if __name__ == "__main__":
    main()
 
You must log in or register to reply here.
Back
Top