CPU limit for jails under ULE scheduler

[kostjn]

Hello all!
I`m rewrire original cdjones patch for cpu limit for jail under ULE scheduler.
So, this work simple.
We count cpu usage for all jails, and if jail use cpu more than have shared cpu, we move his threads to IDLE queue.
Jailed thread can use all avaliable cpu time, if system has avaliable cpu.
If system under heavy load, jailed thread can`t use cpu long as ratio (shared cpu for jail/ all shared cpu) < (estimate usage cpu for jail / all usage cpu) .
Unjailed thread are not subject to this regime.
Interactive thread also are not subject to this regime.
Add 2 sysctl
kern.sched.total_sched_shares - total count shares cpu in system, increase if we have more cpu
kern.sched.flush_estcpu_interval - flush estcpu interval in ticks, default is 2560 = 2 * 128 * 10, NCPU*stathz*sec, increase if we have more cpu
For use cpu limit, you need use flag -S NSharedCPU in /usr/sbin/jail program.
My example jail -S100 /usr/jails/root/ root.kostjn.pht 192.168.0.245 /bin/csh

I`m tested this under 10 simultaneous process in jail and in main system. test program is infinity cycle an 8 core xeon, use RELENG_7.
First run process in jail, and after in main system.
This one process tracking cpu usage
Jail

root 1052 0.0 0.0 3692 784 p1 RJ 7:38PM 0:00.39 /test.o
root 1052 21.2 0.0 3692 784 p1 RJ 7:38PM 0:02.40 /test.o
root 1052 35.6 0.0 3692 784 p1 RJ 7:38PM 0:04.40 /test.o
root 1052 47.5 0.0 3692 784 p1 RJ 7:38PM 0:06.41 /test.o
root 1052 39.9 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 33.2 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 27.6 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 22.9 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 19.0 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 15.8 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 13.0 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 10.8 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /test.o
root 1052 8.9 0.0 3692 784 p1 RJ 7:38PM 0:06.62 /tes

Main system

root 1088 14.9 0.0 3692 780 p0 R 7:38PM 0:01.57 /root/test.o
root 1088 30.8 0.0 3692 780 p0 R 7:38PM 0:03.60 /root/test.o
root 1088 43.8 0.0 3692 780 p0 R 7:38PM 0:05.60 /root/test.o
root 1088 51.0 0.0 3692 780 p0 R 7:38PM 0:07.25 /root/test.o
root 1088 50.8 0.0 3692 780 p0 R 7:38PM 0:08.28 /root/test.o
root 1088 49.1 0.0 3692 780 p0 R 7:38PM 0:09.21 /root/test.o
root 1088 48.1 0.0 3692 780 p0 R 7:38PM 0:10.24 /root/test.o
root 1088 46.2 0.0 3692 780 p0 R 7:38PM 0:11.17 /root/test.o
root 1088 42.9 0.0 3692 780 p0 R 7:38PM 0:11.95 /root/test.o

So we see, that after run in main system, jailed process can`t usage cpu.

I`m don`t have big expirience in kernel programming, consequently best if you see source code.
Please test this patch and post this you restult or all problem.
This is initial version, without tune jail parameter in runtime.
Thank.
Original cdjones cpu and memory limit patch http://wiki.freebsd.org/JailResourceLimits

[kostjn]

Patch for RELENG7 http://kostjn.spb.ru/patch-jail-cpu-limit-7RELENG
Patch for RELEASE71 http://kostjn.spb.ru/patch-jail-cpu-limit-71RELEASE

[DutchDaemon]

*modded up*

[kostjn]

Hi.
I`m rewrite jail limit patch under CURRENT.
New patch limited CPU, memory, filedesc, process.
And allow change limit on the fly.

================================================== =========================
How to use.
================================================== =========================
Build
cvsup CURRENT
cd /usr/src
patch -p0 < patch-jail-limit-8CURRENT
make buildkernel
make buildworld
make installkernel
reboot
make installworld

Create new entry in login.conf, for example class jail128
jail128:\
:cputime=10:\
:memoryuse=128M:\
:maxproc=256:\
:openfiles=1024:\
:tc=default:
Cputime is percent on 1 core.
Openfiles is sum filedesc for all proc in jail.

Create new jail.
...
Add in /etc/rc.conf
jail_test_flags="-Ljail128"

Run new jail
/etc/rc.d/jail start test

================================================== =========================
Sysctl
================================================== =========================
[root@book ~]# sysctl security.jail.limit
security.jail.limit.enable: 1
security.jail.limit.memory_exceed_kill: 0
[root@book ~]# sysctl -d security.jail.limit
security.jail.limit: Jail limit
security.jail.limit.enable: Enable jail limit
security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee
d memory limit


================================================== =========================
Jset and Jget
================================================== =========================
jset and jget is program for set new jail limit and get current limit
Example
[root@book ~]# cat /etc/rc.conf | grep jail2
jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10"
jail_jail2_rootdir="/usr/jails/jail2/"
jail_jail2_hostname="jail2.book.pht"
jail_jail2_interface="re0"
jail_jail2_ip="192.168.200.22"
jail_jail2_flags="-Ljail64"
[root@book ~]# /etc/rc.d/jail start jail2
Configuring jails:.
Starting jails: jail2.book.pht.
[root@book ~]# cd ~kostjn/

[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65

[root@book /home/kostjn]# ./jset.o 1 jail2048
Set new jail limits, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
[root@book /home/kostjn]# ./jget.o 1

Jail limits and rusage, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65

You see that new limit is set.

================================================== =========================
Test
================================================== =========================
Cpu limit
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Script
[root@book /home/kostjn]# cat test.sh
#!/bin/sh
for i in `jot 8 1`; do cpuset -l0 jexec 1 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 2 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 3 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 4 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 5 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 6 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 7 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 8 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 9 /a.out & done

cpuset -l0 jexec 10 /a.out &

Set class for all jail.

[root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done
Set new jail limits, jid = 1
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 2
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 3
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 4
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 5
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 6
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 7
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 8
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 9
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 10
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024

[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat cpu.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
int64_t i,j=0;
char *s;
for (;;){
}

}

Run test.sh

Result
top
last pid: 3513; load averages: 70.87, 37.58, 16.40 up 0+00:44:02 14:19:46
185 processes: 74 running, 111 sleeping
CPU: 49.9% user, 0.0% nice, 0.0% system, 0.2% interrupt, 49.9% idle
Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free
Swap: 4044M Total, 4044M Free

PID JID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN
3502 10 root 1 97 0 1480K 1244K CPU0 0 0:13 8.79% a.out
3474 6 root 1 97 0 1480K 1244K RUN 0 0:04 4.69% a.out
3431 2 root 1 96 0 1480K 1244K RUN 0 0:03 4.30% a.out
3454 4 root 1 97 0 1480K 1244K RUN 0 0:03 4.05% a.out
3422 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.86% a.out
3482 7 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out
3447 3 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out
3429 1 root 1 96 0 1480K 1244K RUN 0 0:03 3.66% a.out
3485 8 root 1 97 0 1480K 1244K RUN 0 0:05 3.56% a.out
3424 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.56% a.out
3464 5 root 1 97 0 1480K 1244K RUN 0 0:02 3.56% a.out
3438 2 root 1 96 0 1480K 1244K RUN 0 0:03 3.47% a.out
3494 9 root 1 96 0 1480K 1244K RUN 0 0:03 3.27% a.out
3497 9 root 1 97 0 1480K 1244K RUN 0 0:05 3.17% a.out
3433 2 root 1 96 0 1480K 1244K RUN 0 0:03 2.88% a.out
3428 1 root 1 96 0 1480K 1244K RUN 0 0:02 2.88% a.out
3487 8 root 1 97 0 1480K 1244K RUN 0 0:04 2.78% a.out


We see that jail 10 (1 thread), used ~10 % cpu under heavy load.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resourse compute
[root@book /home/kostjn]# ./jset.o 1 jail64
Set new jail limits, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
[root@book /home/kostjn]#
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# apachectl stop
/usr/local/sbin/apachectl stop: httpd stopped
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resource limit

[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat mem.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
int64_t i,j=0;
char *s;
for (i=0; i < 1000 ;i++){
s = malloc(100000 * sizeof(char));
}
sleep(1000);
}

[root@jail1 /]# cc mem.c && ./a.out &
[1] 1320
[root@jail1 /]# ls
bash: fork: Cannot allocate memory
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
We see that jail exceed memory limit. And new fork, mmap syscall not
permitted.
If you set sysctl

[root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1
security.jail.limit.memory_exceed_kill: 1 -> 1
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# ./a.out
Killed: 9
[root@jail1 /]# exit
[root@book /home/kostjn]# tail -n 1 /var/log/messages
May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e
xceed memory limit

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
If you attempt set nonexisten class, limit set to infinity.

[root@book /home/kostjn]# ./jset.o 1 jail123
Set new jail limits, jid = 1
Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223
372036854775807


================================================== =========================
Problem
================================================== =========================
If you have problem in this patch.
Add to kernel config
options KTR
options KTR_ENTRIES=1024
options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ| KTR_LOCK|KTR_CONTENTIO N)
options KTR_MASK=KTR_JAIL
options KTR_CPUMASK=0x3
options KTR_VERBOSE
options PRINTF_BUFR_SIZE=128

Rebuild kernel.
Reboot.
Set sysctl
sysctl debug.ktr.mask=65536
and check /var/log/messages

[vivek]

Thanks for posting this. Have you tested it with 72RELEASE?

[kostjn]

Thanks for posting this. Have you tested it with 72RELEASE?

This patch will not work in 7-STABLE.
The reason in that in CURRENT developers have altered a code jail, new system calls and so on have been added.

[Koduc]

To kostjn: thx for your staff.

Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?

[kostjn]

To kostjn: thx for your staff.

Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?
Yes, certainly.
But now, I am engaged in improvement of a patch for CURRENT.
FreeBSD 8 leaves already soon, therefore I do not see sense to transfer a patch to 7 branch.