Ollama

[cracauer@]

My posts are derived from NapoleonWils0n and his projects on podman with cuda and cracauer@ saying it's possible to run cuda in the linuxlator.

Of course for the Linuxulator you also need to compile llama.cpp into Linux binaries.

But I can imagine making a port linux-llama-cpp with CUDA support.

 

[grandpa]

Of course for the Linuxulator you also need to compile llama.cpp into Linux binaries.

But I can imagine making a port linux-llama-cpp with CUDA support.

That would be nice yes. It would require deep knowledge of the uvm modification and how the linuxlators translation layers work so I can't do a port like that.

Besides I have 3 other ports committed and not yet found a reviewer with some spare time.

Also the maintenance when Nvidia changes something that affects the uvm modification would probably be a nightmare.

I just got curious since I don't normally run the linuxlator so there was a learning opportunity for me when I bulldozed this through.

/grandpa

 

[tingo]

you can use https://github.com/AlexsJones/llmfit it is somewhat nice for this purpose

and it downloads the models to a different place (of course). So now I need to tell llama-server how to deal with that.

 

[tingo]

for 16GB card like yours , and 32GB of ram you can maybe go with (with quants ofc)

for best knowledge and quality replies (speed tradeoff)
Qwen3.8-27B

I tried that one, but couldn't figure out how to specify quants that made llama-server run it instead of throwing an "out of memory" error.

 

[gergeskirolos59]

I tried that one, but couldn't figure out how to specify quants that made llama-server run it instead of throwing an "out of memory" error.

why use more space? , download q4 quants not fp16 then try to tell llama to quant it

 

[DanielBowman]

I use Ollama because it integrates nice with Home Assistant. Which is the ideal usage for it having mere 16GB of nVidia VRAM (bought on cheap a year ago, yey).
For tinkering llama.cpp is probably better/faster.
NotebookLM is for full GUI experience, ease of use, manily with huggingface format of models (you can convert them for ollama quite easily, I heard but not sure if you get 1:1 efficiency).

Just today I read about new kind of models, tuned not for text or chat but for decision making and transparent probability readings. That would be HUUUGE. It would diminish the tendency to hallucinate. And it is orders of magnitudes faster and cheaper, especially for output. Such is the claim.
No idea if you can use it with Ollama etc., though.

The new model is now famous (it changes everything, you guys - famous words which makes me NOT click)
JEV

Apparently access is on waiting list and you feed it in some formatted json with question/problem and corresponding options. It assigns probability to these options. i.e. not a chatbot.
Possibly LLM can use this result to form an answer or an agent can do a tool call etc.
So far it lacks in mathematics (e.g. numbers comparisons ???) and other rather exact matters. Well, exact matter can be done by exact algorithm, anyway...like agentic tool call for python or shell script.

Well, if this works, I see it sitting on SOC being fed logs from SIEM, without bankrupting yourself through token usage or burning out human eyes and brains. And the structured inputs can be put together in advance with LLM and JEV too. Outputs with high threat probability can be enhanced by LLM fluff and send to your eyes to decide (or maybe another JEV) ad infinitum, until it reaches Mediocrites