Can FreeBSD run LLM locally?

To answer the topic, yes.

Easiest way to get started is to install ollama from ports. That'll give you either a cli or a web interface. With your setup you should be able to run local AI well using vulkan.

If you want to try some agents there are at least claude-code, crush (charmcrush) and qwen-code in ports.

/grandpa
 
How about vulkan backend?
It should work but as I said the optimizations/improvements present for Linux are trailing behind for FreeBSD - you get 5-15% performance disadvantage and maybe more if new feature gets developed. Currently the best bet is Ubuntu 24+. They have some kind of freundshaft with AMD.
I really don't know what advantages would FreeBSD bring on this box, besides security (Ubuntu is slacking with the latest TCP stack CVE kerfuffle).
 
the optimizations/improvements present for Linux are trailing behind for FreeBSD - you get 5-15% performance disadvantage and maybe more if new feature gets developed.
I'm curious on specifics; I figured AI to be userspace and can improve how it interfaces with hardware, before changing driver/kernel stuff to handle a workload that's afaik not unique (games do 3D/textures, mining is compute on VRAM; what's AI if not compute that's been done scientific, mining, or folding?)

I'd also argue FreeBSD's base can likely process faster than OS-specific "optimizations" (games I ran on Wine ran better on FreeBSD vs rolling Linux/Mesa dev)
 
With nvidia, yes.
But use Linux jail for it. I use Debian 13. Linux jail gives you 1) CUDA 2) isolation 3) far better software support e.g. more agents and mcp tools.

I booted the REA tools over Qwen Code connected to llama.cpp running Qwen 3.5, all running in a jail under limited account and no sudo installed. I had to work around some compatibility issues in setting up Ghidra in a jail, otherwise, it was pretty much lean setup. Gave it Super Metroid ROM and told it to figure out the main character state machine for movement in all conditions, and write me an abstract C implementation for it. It chugged on it for 15 minutes while I was doing something else, and spat it out. It called various basic CLI tools in the jail such as strings, xxd, find etc which may have different switches on FreeBSD and thus MCP servers should be adapted to FreeBSD which is additional work, it decompiled the ROM with Ghidra, and it used gcc to test its implementation on the go, so in the case of compiler tools some differences again might be under the hood.

So my strong recommendation is to run all of the stuff under jailed Linux, in case of nvidia GPU.
 
Locally like entirely offline? It would be interesting.
The local models got pretty good. And with 120GB vRAM you can do a lot. Slowish, not frontier cloud model "smart" but free tokens, no limit. Plus on the AMD setup it does not consume Gigawatts like the gaming cards.
If you use through OpenWebUI or some Agent, you can even employ searxng for anonymous search on the Interent to get the facts straight.
 
The local models got pretty good. And with 120GB vRAM you can do a lot. Slowish, not frontier cloud model "smart" but free tokens, no limit. Plus on the AMD setup it does not consume Gigawatts like the gaming cards.
If you use through OpenWebUI or some Agent, you can even employ searxng for anonymous search on the Interent to get the facts straight.
It could be a shell that listens to human commands. There should be a howto for setting up all requirements to learn it.
 
It could be a shell that listens to human commands. There should be a howto for setting up all requirements to learn it.
It could and there is. Youtube is full of howtos.
For fully local smartish Alexa:
Voice Assistant PE -> Home Assistant -> Hermes Integration -> Hermes Agent -> Local LLM.....and back.
That way you have voice control and feedback (or chat through app). Home Assistant can play music or whatever. And Hermes remembers your choices, can search web for you etc.
Don't have to worry about Bezos listening to you or collect telemetry. Only risk is prompt injection, so Hermes babbles out your data or nudes from cameras.
 
It could and there is. Youtube is full of howtos.
For fully local smartish Alexa:
Voice Assistant PE -> Home Assistant -> Hermes Integration -> Hermes Agent -> Local LLM.....and back.
That way you have voice control and feedback (or chat through app). Home Assistant can play music or whatever. And Hermes remembers your choices, can search web for you etc.
Don't have to worry about Bezos listening to you or collect telemetry. Only risk is prompt injection, so Hermes babbles out your data or nudes from cameras.
Why all end-consumer products? The only thing that's apparently required is a set of wide binary GPU operations. (I don't really believe it. Those operations can be distributed over networks and don't reallly need to directly come from real central hardware)
A learning model with a certain level is a binary blob that can be added and upgraded with more data. If it does its job good enough It doesn't really need to learn new things...
 
Why all end-consumer products?
I don't get your gist, sorry.
It is all end-consumer products for privacy and availability reasons....and integrity.
You can distribute the compute over many boxes, local and/or cloud. I don't because electricity costs - I have just one low wattage AMD box.

It needs to learn news for example or by comparing Internet search info you can correct for "hallucinations". Also your preferences need to be managed outside the model. You can re-fine/re-train the model but it is very compute intensive. Better let agent like Hermes with memory to inject those.
 
I don't get your gist, sorry.
It is all end-consumer products for privacy and availability reasons....and integrity.
You can distribute the compute over many boxes, local and/or cloud. I don't because electricity costs - I have just one low wattage AMD box.

It needs to learn news for example or by comparing Internet search info you can correct for "hallucinations". Also your preferences need to be managed outside the model. You can re-fine/re-train the model but it is very compute intensive. Better let agent like Hermes with memory to inject those.
Alexa and Home Assistant are end solutions and don't have much to do with a locally running AI program.
Install some LLM and give it a blob of data that is its preset intelligence. If it gets English, it should be able to report the free diskspace of all computers on your LAN it has access to when asked on a CLI in text.

A local AI system can't be a single independent program?
 
A local AI system can't be a single independent program?
Can be, if terminal only.
For SpeechToText and TextToSpeech you need other programs. And another one to provide gateway to the IoT realm.
Home Assistant solves the gateway, TTS and STT parts.
Hermes solves the memory part.
But you can communicate with LLM (e.g. Ollama) directly from HomeAssistant...or tty.
 
Can be, if terminal only.
For SpeechToText and TextToSpeech you need other programs. And another one to provide gateway to the IoT realm.
Home Assistant solves the gateway, TTS and STT parts.
Hermes solves the memory part.
But you can communicate with LLM (e.g. Ollama) directly from HomeAssistant...or tty.
The default input of any LLM is text. They can do tricks with media data but that's not interesting.
My idea was a local AI service that does usual shell jobs but without needing a detailed specification like a script. "Backup all data related to project <prj> to computer a.b.c.d with maximal compression. Only operate during idle time and free ethernet."
This should not require any central online service.

To find out where that specific backup has been placed, just ask where it is...
 
This should not require any central online service.
What do you mean by online and central?
It can run all on your one machine. LLM + agent. No online, i.e. cloud needed. Central only in a sense you need computer powerful enough to run the model. You can put more services on it, if you want but you then restrict your RAM usage which could be used for running the model, which needs bulk size, meanwhile your services can run elsewhere.
 
What do you mean by online and central?
It can run all on your one machine. LLM + agent. No online, i.e. cloud needed. Central only in a sense you need computer powerful enough to run the model. You can put more services on it, if you want but you then restrict your RAM usage which could be used for running the model, which needs bulk size, meanwhile your services can run elsewhere.
Online and central like you need some commercial AI service while having local resources to get the same results. Local is a sneaky false term that I see on Youtube. Offline, it should be. In this context "local" is still Chatgpt/Claude,whatever. Even if that processing capacity would somehow be needed, it could still be torrent/p2p based without a corporation involved.
 
Online and central like you need some commercial AI service while having local resources to get the same results. Local is a sneaky false term that I see on Youtube. Offline, it should be. In this context "local" is still Chatgpt/Claude,whatever. Even if that processing capacity would somehow be needed, it could still be torrent/p2p based without a corporation involved.
I don't know what Youtubes you have been watching. In my youtubes Local means NOT Cloud/Corporate, i.e. ChatGPT/Claude etc..
LLM runs on your HW, totally offline if need be.
That is the setup OP asked about.
What would you run over torrent/p2p exactly?
 
I don't know what Youtubes you have been watching. In my youtubes Local means NOT Cloud/Corporate, i.e. ChatGPT/Claude etc..
LLM runs on your HW, totally offline if need be.
That is the setup OP asked about.
What would you run over torrent/p2p exactly?
Not convinced. Not that I'm searching and trying a lot with AI I but haven't seen a independent system that doesn't rely on remote data or closed code to function. Does the concept exist in software, emulated? It should. It contains no particular crucial technology that people can't have at home.
If the service uses a very large database, it can be distributed over the users instead of OpenAI and friends.

I would be willing to pay reasonably for a system that listens to commands like the example I proposed. That's a piece of software tech, and static for amd64 in my case. Quite sure I don't need 20 live datacenters for it.
 
but haven't seen a independent system that doesn't rely on remote data or closed code to function
You can see even in this thread. You can have open weight model LLM, utilizes by open source llama.cpp installed on Ubuntu Server, also open source. All in your home.
Maybe you can combine the LLMs with friends to have even bigger LLM/database of weights, maybe on par with OpenAI etc. if you have enough friends. Not sure it will be fast enough over TCP/UDP tho.
However the setup OP asks about is most likely handy enough to meet your demands i.e. listen to commands you proposed. Fully offline and opensource bar BIOS and maybe some drivers.
 
You can see even in this thread. You can have open weight model LLM, utilizes by open source llama.cpp installed on Ubuntu Server, also open source. All in your home.
Maybe you can combine the LLMs with friends to have even bigger LLM/database of weights, maybe on par with OpenAI etc. if you have enough friends. Not sure it will be fast enough over TCP/UDP tho.
However the setup OP asks about is most likely handy enough to meet your demands i.e. listen to commands you proposed. Fully offline and opensource bar BIOS and maybe some drivers.
Weights? I think of a program that interprets human language and gets the idea of given sysadmin jobs without explaining every detail.
I believe if this worked, it would have existed, as 1 executable file. A complex construct is a clear sign we're not there yet. Not on 1 personal computer, at least.
 
short answer: yes you can.
you have ollama and llama.cpp, i tested ollama and its work fine but i think ollama have lack of gpu support in freebsd and what i read llama.cpp is more prepared and can improve a bit with custom settings. and if you want a client interface like gemini or chatgpt... mistral... etc... there is openweb ui is better if you use some kind of linux compatibility for that.
 
1 executable file
That is a very unUNIXly demand.

on 1 personal computer

UGREEN showed one box like that NAS/Home Voice controlled Assistant/Ai in one box. Not a personal computer, tho.

But as Moore's law goes it will all end up in your phone in near future. Maybe even in one binary blob to suit your needs.
Hold tight.

In the meantime, in reality, Strix Halo is the cheapest and most private option to get that Ai functionality.
 
Back
Top