jails Automatic rebalancing cpuset for jails

A

An-tonio

Hi,

I have searched on this forum and on internet without success so here is my question.

I love jails and his flexibility but for a "hosting" service with clients on jails like vps I would like to limit the max number of cpu cores each jail can utilize on the server. I know about cpuset and I think it's the way to go for now (I did not find the way to do it with rctl). With cpuset It's possible to assign certain specific cores to jails, so the problem can be solved but it have to be done manually. Do you know if exist some daemon or service that "rebalance" the cpu cores of each jail automatically ? This daemon would set the cpu cores for each jail (max limited) in a periodic interval to prevent multiple jails utilize the same cores and try to balance the best usage of whole cpus available on host.

I know that proxmox do something similar with his pvestatd service for lxc containers and modify the assigned cores in cgroups cpusets in a periodic interval.


Code: 
 cpuset
 -----------------------
 jail1:      2         7
 jail2:    1   3 4 5
 jail3:  0 1     4   6
 -----------------------
 -----------------------
 jail1:  0 1
 jail2:  0       4 5 6
 jail3:      2 3     6 7
 -----------------------

Thanks.
 
I'm not a specialist, so i might something stupid.
Thought the O.S. assign tasks to the cores which are at a certain time in idle state for a period of time ?
 
Yes, when all jails use all cores on host it's like that, freebsd (and any OS) balance the load on all cpus automatically. But I want to restrict each jail to a max number of permitted cores. Manually I can assign each jail a set of cores but when the sum of all jails permitted cores are higher than total host cores it's interesting to do this assignment automatically and rotate assigned cpus to each jails in the best optimal way.
 
Yes, I use cpuset to set manually limited cores for each jail but I'm searching for an automated way of doing it based on the config of each jails and to "rebalance" the assigned cores in the best way in a periodic interval to optimize host cpu usage.
 
can't you use pcpu/jail with rctl ?
and give them 100 or 200 or 300 or whatever number of cores you want for each jail
or even a combination ncpu and nthr
 
covacat said:
can't you use pcpu/jail with rctl ?
and give them 100 or 200 or 300 or whatever number of cores you want for each jail
or even a combination ncpu and nthr
Click to expand...
I too would like to know whether this works. Please keep us posted :)
 
Not immediately directly related , but i use a resource limit on the jail in which i run poudriere. I limit memory usage (swap) against starvation,
Code: 
/usr/bin/rctl -a jail:pj:memoryuse:deny=13G

Note : in the mailinglists i found some references regarding jails/cpusets so work is there currently being done.
 
If you use cpuset, watch your host cpu topology ... so if you assign e.g. 2 cores to the jail they should be in the same group (core + thread) - to "rebalance" then cpuset is not the right solution because with cpuset you can only restrict the cpus that a jail uses; so in case to "rebalance" you need to restart the jail with a new cpuset.

for rebalancing I suggest pcpu.
 
I did some test with rctl pcpu and it's not clear for me how it work and how to get proper results. I did some rapid benchmarks with 7z b and I get always solid results with cpusets (and I can verify full cpu usage of each core assigned).

If I use rctl and set pcpu to 300 for 3 cpu usage (in theory) the results are differents from cpuset with 3 cores assigned. While the test, the cpu usage on host server vary a lot from all cores at full to all cores at near 0% activity or 50% each.
The global test are faster with rctl pcpu set to 300 than with cpuset with 3 cores.

So, maybe I did not found the way to get proper results with rctl or it's no possible actually to get the "rebalancing" cpu for hosted jails ?
 
Maybe it is set for "any" 3 cpu's, idont know the details, and maybe it's up to the kernel ule scheduler ?
In symmetric multiprocessing task are divided under the cores.
 
might be worth hitting the mailing lists(s) on this one. The kernel developers don't usually hang around this forum (for understandable reasons, if I may say so).
 
Alain De Vos said:
I think there is nothing wrong in asking questions to enlighten the current status.
Click to expand...
not sure whether that is directed to my previous post but I certainly fully agree. I didn't mean to dismiss anyone's statement/question or dismiss an individual as a whole.
Experience just shows that these type of questions usually get better/quicker/more-accurate answers on the corresponding mailing lists.
 
You are right jbodenmann, maybe it can be better to ask in the mailing list.
I thought it might not be so complicated.
 
An-tonio said:
You are right jbodenmann, maybe it can be better to ask in the mailing list.
I thought it might not be so complicated.
Click to expand...
I think you should write a little wrapper before starting containers. At least in the base system you will not find automation. If you have nothing against the shell, you can see how the CBSD does it.
The scripts looks for the most unused cores and assigns it to the container. For example, the vcores map of my AMD Ryzen 9 3900X 12-Core Processor after creating two containers ( cpu limit: '1' for first jail / '3' - for second )

cbsd jcreate jname=small cpu=1 fsquota=10g vmemoryuse=512m runasap=1
cbsd jcreate jname=midd cpu=3 fsquota=10g vmemoryuse=512m runasap=1


1639872279693.png
 
I can +1 for looking at how sysutils/cbsd is doing this. It's a great utility I've been using for many years now. Resource management worked as per my expectations.

Actually, depending on your use case you might just wanna start using systutils/cbsd and be done with it :D
 
  • Like
Reactions: Ole
You must log in or register to reply here.
Back
Top