jails llama-cli tok/s increase by 50% when running it in a tight jail

loveydovey

loveydovey

Running a local LLM model with llama-cli gets a 43% boost in tok/s running on CPU if I run llama inside a jail without anything much.

I am curious why. What would slow it down running on the host as is?
 
A jail adds no virtualization layer, so it can't speed anything up by itself. The difference almost certainly comes from the environment. Most likely: llama-cli defaults its thread count to the number of visible logical CPUs. If the jail is restricted via cpuset, llama picks fewer threads, and on an SMT machine one thread per physical core beats one per logical core for token generation, which fits a ~40% delta well. Try an explicit -t <physical cores> on the host. Also check that both runs use the identical binary and an otherwise idle machine.
 
I did run it with -t limited to the physical core count. So it's not that specifically, unless there's some OTHER hidden option that needs to be also set that is related to this. Or maybe -t is not working as intended on the host, although I doubt that.

I just reran it with and without a jail back to back, and I'm getting 10% faster tok/s with jail. It may vary, I guess, but there's a definitive bump to tok/s when run inside the jail, which is a surprise.

So I'm still curious. Maybe there's a way to figure out why?
 
10% with the thread count pinned sounds like scheduler affinity rather than thread count. If your jail has a cpuset, the threads stay pinned to those CPUs (no migrations, warm caches, and no two threads landing on the same physical core). You can test this on the host: check the jail's set with cpuset -g -j <jid>, then run cpuset -l <same CPUs> llama-cli ... on the host. If that closes the gap, it was affinity. For measuring, llama-bench with several repetitions is more reliable than single llama-cli runs.
 
You must log in or register to reply here.
Back
Top