Ollama

hmm, so far I can't even get it to run
Code: 
tingo@locaal:~ $ llama-server \
>         --host `hostname` \
>         --port 8080 \
>         --ctx-size $((64 * 1024)) \
>         --temp 0.6 \
>         --top-p 0.95 \
>         --top-k 20 \
>         -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L
Downloading mmproj-Qwen_Qwen3.5-27B-bf16.gguf ────────────────────── 100%
Downloading Qwen_Qwen3.5-27B-Q6_K_L.gguf ─────────────────────────── 100%
12.34.224.670 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
12.34.224.674 I device_info:
12.34.225.223 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
12.34.225.244 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
12.34.225.307 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
12.34.225.315 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
12.34.226.729 I srv          init: running without SSL
12.34.226.755 I srv          init: using 11 threads for HTTP server
12.34.227.973 I srv         start: binding port with default address family
12.34.240.500 E srv         start: couldn't bind HTTP server socket, hostname: locaal, port: 8080
12.34.240.503 I srv    operator(): operator(): cleaning up before exit...
12.34.241.463 E srv  llama_server: exiting due to HTTP server error
It shouldn't be necessary to run it as root, it uses an unprivileged port.
 
balanga - the port works fine so install and in one terminal do ollama serve, go to ollama library to find the name of a model and then in another terminal do ollama run <name>.

tingo - it's needs an ip or "localhost". If you want to serve on LAN ip go with the LAN ip or 0.0.0.0 to bind to all ips in the machine.

(For more fun have a look at Cracauers other thread posts #10 and #30. #10 is with vulkan and #30 is with cuda and linuxlator.)

/grandpa
 
aha!
it's not an error, the machine is named locaal.local.
So a bit of change
Code: 
tingo@locaal:~ $ llama-server \
        --host `hostname`.local \
        --port 8081 \
        --ctx-size $((64 * 1024)) \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q6_K_L
0.00.659.799 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.659.803 I device_info:
0.00.659.915 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
0.00.659.930 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
0.00.659.994 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
0.00.660.000 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.660.023 I srv          init: running without SSL
0.00.660.043 I srv          init: using 11 threads for HTTP server
0.00.660.128 I srv         start: binding port with default address family
0.00.883.847 I srv  llama_server: loading model
0.00.883.856 I srv    load_model: loading model '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q6_K_L.gguf'
0.02.907.520 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 1161.02 MiB
0.02.907.560 I common_init_result: fitting params to device memory ...
0.02.907.561 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
ggml_vulkan: Memory allocation of size 1350860800 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
0.12.337.483 E llama_model_load: error loading model: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
0.12.337.495 E llama_model_load_from_file_impl: failed to load model
0.12.337.498 E common_init_from_params: failed to load model '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q6_K_L.gguf'
0.12.337.506 E srv    load_model: failed to load model, '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q6_K_L.gguf'
0.12.337.508 I srv    operator(): operator(): cleaning up before exit...
0.12.337.776 E srv  llama_server: exiting due to model loading error
tingo@locaal:~ $
Now I just need to find a model that fits into my GFX card ( 16GB).
 
Hmm, is there a quicker way of finding out if a given model fits into your GFX card memory than downloading and trying?
Code: 
tingo@locaal:~ $ llama-server \
        --host `hostname`.local \
        --port 8081 \
        --ctx-size $((64 * 1024)) \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        -fit off -hf bartowski/Qwen_Qwen3.5-27B-GGUF:Q3_K_L
0.00.704.983 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.704.987 I device_info:
0.00.705.100 I   - Vulkan0 : NVIDIA GeForce RTX 5060 Ti (16557 MiB, 16071 MiB free)
0.00.705.115 I   - CPU     : CPU (32659 MiB, 32659 MiB free)
0.00.705.178 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : OPENMP = 1 | REPACK = 1 | 
0.00.705.184 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.705.208 I srv          init: running without SSL
0.00.705.228 I srv          init: using 11 threads for HTTP server
0.00.705.314 I srv         start: binding port with default address family
0.00.706.762 I srv  llama_server: loading model
0.00.706.770 I srv    load_model: loading model '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_L.gguf'
ggml_vulkan: Memory allocation of size 1042944000 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
0.05.576.883 E llama_model_load: error loading model: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
0.05.576.894 E llama_model_load_from_file_impl: failed to load model
0.05.576.900 E common_init_from_params: failed to load model '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_L.gguf'
0.05.576.906 E srv    load_model: failed to load model, '/home/tingo/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3.5-27B-GGUF/snapshots/d7b113c40283f4d99f4eb0ec20d126ad653cc736/Qwen_Qwen3.5-27B-Q3_K_L.gguf'
0.05.576.909 I srv    operator(): operator(): cleaning up before exit...
0.05.577.746 E srv  llama_server: exiting due to model loading error
 
Basically you will need to read metadata from the model for layers and trained context window, decide if/how much you need to quantize your caches and then try to optimize the size of the context widow to make it fit in your RAM/NVRAM setup. To complicate matters it is possible to offload layers into RAM at a performance loss but this will allow you to run models that are larger than your NVRAM. If your context window drops too low you will need to allow the model to "forget" a percentage of the context so it can roll on without hitting the context ceiling.

/grandpa
 
balanga said:
Well I'm definitely a beginner, or rather I will be when I finally begin.
Click to expand...
lamma.cpp is faster than olama + ollama is build on top of lamma.cpp so its a wrapper for lamma.cpp
lamma.cpp is the way even for beginners. You can easily use hugging face models ( just download an use them) and if you use ollama and downloaded models from there you can only use with ollama.
ollama better model swapping but with lamma.cpp you need to use lamma-swap.
learn lamma.cpp even if it takes longer to understand and figure it out it will be way more beneficial than ollama.
P.s. i never managed to run ollama with larger models than 25b on my 32gb nvidia gpu ( volta ) always crashing.
 
balanga said:
Anyone familiar with Ollama ?

Opinions?
Click to expand...
I use Ollama because it integrates nice with Home Assistant. Which is the ideal usage for it having mere 16GB of nVidia VRAM (bought on cheap a year ago, yey).
For tinkering llama.cpp is probably better/faster.
NotebookLM is for full GUI experience, ease of use, manily with huggingface format of models (you can convert them for ollama quite easily, I heard but not sure if you get 1:1 efficiency).

Just today I read about new kind of models, tuned not for text or chat but for decision making and transparent probability readings. That would be HUUUGE. It would diminish the tendency to hallucinate. And it is orders of magnitudes faster and cheaper, especially for output. Such is the claim.
No idea if you can use it with Ollama etc., though.
 
I use Ollama because it integrates nice with Home Assistant. Which is the ideal usage for it having mere 16GB of nVidia VRAM (bought on cheap a year ago, yey).
For tinkering llama.cpp is probably better/faster.
NotebookLM is for full GUI experience, ease of use, manily with huggingface format of models (you can convert them for ollama quite easily, I heard but not sure if you get 1:1 efficiency).

Just today I read about new kind of models, tuned not for text or chat but for decision making and transparent probability readings. That would be HUUUGE. It would diminish the tendency to hallucinate. And it is orders of magnitudes faster and cheaper, especially for output. Such is the claim.
No idea if you can use it with Ollama etc., though.
 
I use Ollama because it integrates nice with Home Assistant. Which is the ideal usage for it having mere 16GB of nVidia VRAM (bought on cheap a year ago, yey).
For tinkering llama.cpp is probably better/faster.
NotebookLM is for full GUI experience, ease of use, manily with huggingface format of models (you can convert them for ollama quite easily, I heard but not sure if you get 1:1 efficiency).

Just today I read about new kind of models, tuned not for text or chat but for decision making and transparent probability readings. That would be HUUUGE. It would diminish the tendency to hallucinate. And it is orders of magnitudes faster and cheaper, especially for output. Such is the claim.
No idea if you can use it with Ollama etc., though.
 
grandpa said:
Take a look at my scripts as mentioned in #9.

/grandpa
Click to expand...
You are probably using a newer version of llam-server than me, mine does not ssuport lod-mode,. it seems
Code: 
tingo@locaal:~ $ sh bin/run_llama.sh
Pick an AI model to run: 
1)     0.00G   Qwen3.8-27B-IQ2_XS.gguf
2)     0.00G   Qwen_Qwen3.5-27B-Q2_K_L.gguf
3)     0.00G   Qwen_Qwen3.5-27B-Q3_K_L.gguf
4)     0.00G   Qwen_Qwen3.5-27B-Q3_K_M.gguf
5)     0.00G   Qwen_Qwen3.5-27B-Q3_K_S.gguf
6)     0.00G   Qwen_Qwen3.5-27B-Q6_K_L.gguf
7)     0.00G   TheDrummer_Orion-26B-A4B-v1.1-IQ3_XS.gguf
8)     0.00G   mmproj-Qwen3.8-27B-bf16.gguf
9)     0.00G   mmproj-Qwen_Qwen3.5-27B-bf16.gguf
a)     0.00G   mmproj-TheDrummer_Orion-26B-A4B-v1.1-bf16.gguf
0) Exit
Your choice: 1
──────────────────────────────────────────────────────
📊 Analyzing Qwen3.8-27B-IQ2_XS.gguf (0.00 GB)...
🎮 Verifying NVIDIA GPU status...
✅ GPU Status: NVIDIA Graphics Driver (NVIDIA UNIX x86_64 Kernel Module  595.84  Wed Jun 10 21:13:57 UTC 2026) is active.
🧠 Reading model built-in metadata live...
ℹ  Could not read trained context length. Using safe fallback: 32768
ℹ  Could not read layer count. Using baseline fallback: 64 layers.
⚙️  Optimized profile: CTX=32768, NGL=69, BATCH=512, UBATCH=256, CACHE=f16, MODE=mlock
🔄 Context Shifting: Active (Infinite scrolling chat enabled)
💡 Safe Prompt Limit: Do not exceed 32256 tokens in a SINGLE prompt.
──────────────────────────────────────────────────────
error: invalid argument: --load-mode
 
tingo said:
Hmm, is there a quicker way of finding out if a given model fits into your GFX card memory than downloading and trying?
Click to expand...

there are but trust me you will download alot of models anyway to compare their outputs for your use case so just download and try if your bandwidth allows this/unlimited


tools that detect what models you can run are mostly bad because they compare model size aganist vram not context window you will use too making them inacurate but you can use https://github.com/AlexsJones/llmfit it is somewhat nice for this purpose


for 16GB card like yours , and 32GB of ram you can maybe go with (with quants ofc)

for best knowledge and quality replies (speed tradeoff)
Qwen3.8-27B
muse-glimmer
gemma4:26b/31b (gemini is bad in coding do not expect gemma to be much different if that is your use case)

if you appricate speed more than accuracy
Qwen3.5-35B-A3B (it will be so fast)
 
tingo said:
You are probably using a newer version of llam-server than me, mine does not ssuport lod-mode,. it seems
Click to expand...
That's correct. I am using the very latest version since it is possible to do that when running llama-cpp in the linuxlator (with cuda support). It's all in my posts.

There are many thing I need to change in my scripts and in how the llama runs in the linuxlator, right now I am simplifying the scripts and I need to drop llama-cpp from running as chroot in the linuxlator to a normal user.

My posts are derived from NapoleonWils0n and his projects on podman with cuda and cracauer@ saying it's possible to run cuda in the linuxlator.

/grandpa
 
You must log in or register to reply here.
Back
Top