Not very goodMy setup is an AI Max 395 with 128GB RAM.
How about vulkan backend?Not very good
No NPU.
No RocM.
on FreeBSD whatsoever.
AMD works closely with Ubuntu, FreeBSD ignored.
DRM trailing behind Linux optimizations (costing 5 to 15%)
Only pain, no gain. Zfs and jails are irrelevant on this setup as one should not run anything else but llama.cpp there for max memory.
It should work but as I said the optimizations/improvements present for Linux are trailing behind for FreeBSD - you get 5-15% performance disadvantage and maybe more if new feature gets developed. Currently the best bet is Ubuntu 24+. They have some kind of freundshaft with AMD.How about vulkan backend?
I'm curious on specifics; I figured AI to be userspace and can improve how it interfaces with hardware, before changing driver/kernel stuff to handle a workload that's afaik not unique (games do 3D/textures, mining is compute on VRAM; what's AI if not compute that's been done scientific, mining, or folding?)the optimizations/improvements present for Linux are trailing behind for FreeBSD - you get 5-15% performance disadvantage and maybe more if new feature gets developed.
The local models got pretty good. And with 120GB vRAM you can do a lot. Slowish, not frontier cloud model "smart" but free tokens, no limit. Plus on the AMD setup it does not consume Gigawatts like the gaming cards.Locally like entirely offline? It would be interesting.
It could be a shell that listens to human commands. There should be a howto for setting up all requirements to learn it.The local models got pretty good. And with 120GB vRAM you can do a lot. Slowish, not frontier cloud model "smart" but free tokens, no limit. Plus on the AMD setup it does not consume Gigawatts like the gaming cards.
If you use through OpenWebUI or some Agent, you can even employ searxng for anonymous search on the Interent to get the facts straight.
It could and there is. Youtube is full of howtos.It could be a shell that listens to human commands. There should be a howto for setting up all requirements to learn it.
Why all end-consumer products? The only thing that's apparently required is a set of wide binary GPU operations. (I don't really believe it. Those operations can be distributed over networks and don't reallly need to directly come from real central hardware)It could and there is. Youtube is full of howtos.
For fully local smartish Alexa:
Voice Assistant PE -> Home Assistant -> Hermes Integration -> Hermes Agent -> Local LLM.....and back.
That way you have voice control and feedback (or chat through app). Home Assistant can play music or whatever. And Hermes remembers your choices, can search web for you etc.
Don't have to worry about Bezos listening to you or collect telemetry. Only risk is prompt injection, so Hermes babbles out your data or nudes from cameras.
I don't get your gist, sorry.Why all end-consumer products?
Alexa and Home Assistant are end solutions and don't have much to do with a locally running AI program.I don't get your gist, sorry.
It is all end-consumer products for privacy and availability reasons....and integrity.
You can distribute the compute over many boxes, local and/or cloud. I don't because electricity costs - I have just one low wattage AMD box.
It needs to learn news for example or by comparing Internet search info you can correct for "hallucinations". Also your preferences need to be managed outside the model. You can re-fine/re-train the model but it is very compute intensive. Better let agent like Hermes with memory to inject those.
Can be, if terminal only.A local AI system can't be a single independent program?
The default input of any LLM is text. They can do tricks with media data but that's not interesting.Can be, if terminal only.
For SpeechToText and TextToSpeech you need other programs. And another one to provide gateway to the IoT realm.
Home Assistant solves the gateway, TTS and STT parts.
Hermes solves the memory part.
But you can communicate with LLM (e.g. Ollama) directly from HomeAssistant...or tty.
What do you mean by online and central?This should not require any central online service.
Online and central like you need some commercial AI service while having local resources to get the same results. Local is a sneaky false term that I see on Youtube. Offline, it should be. In this context "local" is still Chatgpt/Claude,whatever. Even if that processing capacity would somehow be needed, it could still be torrent/p2p based without a corporation involved.What do you mean by online and central?
It can run all on your one machine. LLM + agent. No online, i.e. cloud needed. Central only in a sense you need computer powerful enough to run the model. You can put more services on it, if you want but you then restrict your RAM usage which could be used for running the model, which needs bulk size, meanwhile your services can run elsewhere.
I don't know what Youtubes you have been watching. In my youtubes Local means NOT Cloud/Corporate, i.e. ChatGPT/Claude etc..Online and central like you need some commercial AI service while having local resources to get the same results. Local is a sneaky false term that I see on Youtube. Offline, it should be. In this context "local" is still Chatgpt/Claude,whatever. Even if that processing capacity would somehow be needed, it could still be torrent/p2p based without a corporation involved.
Not convinced. Not that I'm searching and trying a lot with AI I but haven't seen a independent system that doesn't rely on remote data or closed code to function. Does the concept exist in software, emulated? It should. It contains no particular crucial technology that people can't have at home.I don't know what Youtubes you have been watching. In my youtubes Local means NOT Cloud/Corporate, i.e. ChatGPT/Claude etc..
LLM runs on your HW, totally offline if need be.
That is the setup OP asked about.
What would you run over torrent/p2p exactly?
You can see even in this thread. You can have open weight model LLM, utilizes by open source llama.cpp installed on Ubuntu Server, also open source. All in your home.but haven't seen a independent system that doesn't rely on remote data or closed code to function
Weights? I think of a program that interprets human language and gets the idea of given sysadmin jobs without explaining every detail.You can see even in this thread. You can have open weight model LLM, utilizes by open source llama.cpp installed on Ubuntu Server, also open source. All in your home.
Maybe you can combine the LLMs with friends to have even bigger LLM/database of weights, maybe on par with OpenAI etc. if you have enough friends. Not sure it will be fast enough over TCP/UDP tho.
However the setup OP asks about is most likely handy enough to meet your demands i.e. listen to commands you proposed. Fully offline and opensource bar BIOS and maybe some drivers.
That is a very unUNIXly demand.1 executable file
on 1 personal computer