CPU limit for jails under ULE scheduler

Kernel development, writing drivers, coding, and questions regarding FreeBSD internals.

CPU limit for jails under ULE scheduler

Postby kostjn » 17 Apr 2009, 14:29

Hello all!
I`m rewrire original cdjones patch for cpu limit for jail under ULE scheduler.
So, this work simple.
We count cpu usage for all jails, and if jail use cpu more than have shared cpu, we move his threads to IDLE queue.
Jailed thread can use all avaliable cpu time, if system has avaliable cpu.
If system under heavy load, jailed thread can`t use cpu long as ratio (shared cpu for jail/ all shared cpu) < (estimate usage cpu for jail / all usage cpu) .
Unjailed thread are not subject to this regime.
Interactive thread also are not subject to this regime.
Add 2 sysctl
kern.sched.total_sched_shares - total count shares cpu in system, increase if we have more cpu
kern.sched.flush_estcpu_interval - flush estcpu interval in ticks, default is 2560 = 2 * 128 * 10, NCPU*stathz*sec, increase if we have more cpu
For use cpu limit, you need use flag -S NSharedCPU in /usr/sbin/jail program.
My example
Code: Select all
jail -S100 /usr/jails/root/ root.kostjn.pht  192.168.0.245 /bin/csh


I`m tested this under 10 simultaneous process in jail and in main system. test program is infinity cycle an 8 core xeon, use RELENG_7.
First run process in jail, and after in main system.
This one process tracking cpu usage
Jail
Code: Select all
root    1052  0.0  0.0  3692   784  p1  RJ    7:38PM   0:00.39 /test.o
root    1052 21.2  0.0  3692   784  p1  RJ    7:38PM   0:02.40 /test.o
root    1052 35.6  0.0  3692   784  p1  RJ    7:38PM   0:04.40 /test.o
root    1052 47.5  0.0  3692   784  p1  RJ    7:38PM   0:06.41 /test.o
root    1052 39.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 33.2  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 27.6  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 22.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 19.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 15.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 13.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 10.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052  8.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /tes

Main system
Code: Select all
root    1088 14.9  0.0  3692   780  p0  R     7:38PM   0:01.57 /root/test.o
root    1088 30.8  0.0  3692   780  p0  R     7:38PM   0:03.60 /root/test.o
root    1088 43.8  0.0  3692   780  p0  R     7:38PM   0:05.60 /root/test.o
root    1088 51.0  0.0  3692   780  p0  R     7:38PM   0:07.25 /root/test.o
root    1088 50.8  0.0  3692   780  p0  R     7:38PM   0:08.28 /root/test.o
root    1088 49.1  0.0  3692   780  p0  R     7:38PM   0:09.21 /root/test.o
root    1088 48.1  0.0  3692   780  p0  R     7:38PM   0:10.24 /root/test.o
root    1088 46.2  0.0  3692   780  p0  R     7:38PM   0:11.17 /root/test.o
root    1088 42.9  0.0  3692   780  p0  R     7:38PM   0:11.95 /root/test.o

So we see, that after run in main system, jailed process can`t usage cpu.

I`m don`t have big expirience in kernel programming, consequently best if you see source code.
Please test this patch and post this you restult or all problem.
This is initial version, without tune jail parameter in runtime.
Thank.
Original cdjones cpu and memory limit patch http://wiki.freebsd.org/JailResourceLimits
kostjn
Junior Member
 
Posts: 7
Joined: 17 Apr 2009, 14:21

Postby kostjn » 18 Apr 2009, 07:03

kostjn
Junior Member
 
Posts: 7
Joined: 17 Apr 2009, 14:21

Postby DutchDaemon » 18 Apr 2009, 14:19

*modded up*
User avatar
DutchDaemon
Old Fart
 
Posts: 10467
Joined: 16 Nov 2008, 20:17
Location: The Netherlands

Jail limit for CURRENT

Postby kostjn » 20 May 2009, 15:32

Hi.
I`m rewrite jail limit patch under CURRENT.
New patch limited CPU, memory, filedesc, process.
And allow change limit on the fly.

===========================================================================
How to use.
===========================================================================
Build
Code: Select all
cvsup CURRENT
cd /usr/src
patch -p0 < patch-jail-limit-8CURRENT
make buildkernel
make buildworld
make installkernel
reboot
make installworld


Create new entry in login.conf, for example class jail128
Code: Select all
jail128:\
   :cputime=10:\
   :memoryuse=128M:\
   :maxproc=256:\
   :openfiles=1024:\
   :tc=default:

Cputime is percent on 1 core.
Openfiles is sum filedesc for all proc in jail.

Create new jail.
...
Add in /etc/rc.conf
Code: Select all
   jail_test_flags="-Ljail128"


Run new jail
Code: Select all
/etc/rc.d/jail start test


===========================================================================
Sysctl
===========================================================================
Code: Select all
[root@book ~]# sysctl security.jail.limit
security.jail.limit.enable: 1
security.jail.limit.memory_exceed_kill: 0
[root@book ~]# sysctl -d security.jail.limit
security.jail.limit: Jail limit
security.jail.limit.enable: Enable jail limit
security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee
d memory limit



===========================================================================
Jset and Jget
===========================================================================
jset and jget is program for set new jail limit and get current limit
Example
Code: Select all
[root@book ~]# cat /etc/rc.conf | grep jail2
jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10"
jail_jail2_rootdir="/usr/jails/jail2/"
jail_jail2_hostname="jail2.book.pht"
jail_jail2_interface="re0"
jail_jail2_ip="192.168.200.22"
jail_jail2_flags="-Ljail64"
[root@book ~]# /etc/rc.d/jail start jail2
Configuring jails:.
Starting jails: jail2.book.pht.
[root@book ~]# cd ~kostjn/

[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65

[root@book /home/kostjn]# ./jset.o 1 jail2048
Set new jail limits, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
[root@book /home/kostjn]# ./jget.o 1

Jail limits and rusage, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65


You see that new limit is set.

===========================================================================
Test
===========================================================================
Cpu limit
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Script
Code: Select all
[root@book /home/kostjn]# cat test.sh
#!/bin/sh
for i in `jot 8 1`; do cpuset -l0  jexec 1 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 2 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 3 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 4 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 5 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 6 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 7 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 8 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 9 /a.out & done

cpuset -l0  jexec 10 /a.out &

Set class for all jail.

[root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done
Set new jail limits, jid = 1
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 2
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 3
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 4
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 5
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 6
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 7
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 8
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 9
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 10
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024

[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat cpu.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
        int64_t i,j=0;
        char *s;
        for (;;){
        }

}

Run test.sh

Result
top
last pid:  3513;  load averages: 70.87, 37.58, 16.40    up 0+00:44:02  14:19:46
185 processes: 74 running, 111 sleeping
CPU: 49.9% user,  0.0% nice,  0.0% system,  0.2% interrupt, 49.9% idle
Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free
Swap: 4044M Total, 4044M Free

  PID JID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAN
 3502  10 root          1  97    0  1480K  1244K CPU0    0   0:13  8.79% a.out
 3474   6 root          1  97    0  1480K  1244K RUN     0   0:04  4.69% a.out
 3431   2 root          1  96    0  1480K  1244K RUN     0   0:03  4.30% a.out
 3454   4 root          1  97    0  1480K  1244K RUN     0   0:03  4.05% a.out
 3422   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.86% a.out
 3482   7 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
 3447   3 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
 3429   1 root          1  96    0  1480K  1244K RUN     0   0:03  3.66% a.out
 3485   8 root          1  97    0  1480K  1244K RUN     0   0:05  3.56% a.out
 3424   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.56% a.out
 3464   5 root          1  97    0  1480K  1244K RUN     0   0:02  3.56% a.out
 3438   2 root          1  96    0  1480K  1244K RUN     0   0:03  3.47% a.out
 3494   9 root          1  96    0  1480K  1244K RUN     0   0:03  3.27% a.out
 3497   9 root          1  97    0  1480K  1244K RUN     0   0:05  3.17% a.out
 3433   2 root          1  96    0  1480K  1244K RUN     0   0:03  2.88% a.out
 3428   1 root          1  96    0  1480K  1244K RUN     0   0:02  2.88% a.out
 3487   8 root          1  97    0  1480K  1244K RUN     0   0:04  2.78% a.out


We see that jail 10 (1 thread), used ~10 % cpu under heavy load.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resourse compute
Code: Select all
[root@book /home/kostjn]# ./jset.o 1 jail64
Set new jail limits, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
[root@book /home/kostjn]#
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# apachectl stop
/usr/local/sbin/apachectl stop: httpd stopped
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24


<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resource limit
Code: Select all
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat mem.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
        int64_t i,j=0;
        char *s;
        for (i=0; i  < 1000 ;i++){
                s = malloc(100000 * sizeof(char));
        }
        sleep(1000);
}

[root@jail1 /]# cc mem.c && ./a.out &
[1] 1320
[root@jail1 /]# ls
bash: fork: Cannot allocate memory
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31


<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
We see that jail exceed memory limit. And new fork, mmap syscall not
permitted.
If you set sysctl

Code: Select all
[root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1
security.jail.limit.memory_exceed_kill: 1 -> 1
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# ./a.out
Killed: 9
[root@jail1 /]# exit
[root@book /home/kostjn]# tail -n 1 /var/log/messages
May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e
xceed memory limit


<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
If you attempt set nonexisten class, limit set to infinity.

Code: Select all
[root@book /home/kostjn]# ./jset.o 1 jail123
Set new jail limits, jid = 1
Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223
372036854775807



===========================================================================
Problem
===========================================================================
If you have problem in this patch.
Add to kernel config
Code: Select all
options     KTR
options     KTR_ENTRIES=1024
options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N)
options     KTR_MASK=KTR_JAIL
options     KTR_CPUMASK=0x3
options     KTR_VERBOSE
options     PRINTF_BUFR_SIZE=128

Rebuild kernel.
Reboot.
Set sysctl
Code: Select all
sysctl debug.ktr.mask=65536

and check /var/log/messages
kostjn
Junior Member
 
Posts: 7
Joined: 17 Apr 2009, 14:21

Postby vivek » 27 May 2009, 13:58

Thanks for posting this. Have you tested it with 72RELEASE?
Neither in this world nor elsewhere is there any happiness in store for him who always doubts. If you enjoyed my answer please consider donating some money to FreeBSD foundation @ http://www.freebsdfoundation.org/
User avatar
vivek
Member
 
Posts: 809
Joined: 17 Nov 2008, 08:19
Location: Hyper Space

Postby kostjn » 28 May 2009, 06:09

vivek wrote:Thanks for posting this. Have you tested it with 72RELEASE?


This patch will not work in 7-STABLE.
The reason in that in CURRENT developers have altered a code jail, new system calls and so on have been added.
kostjn
Junior Member
 
Posts: 7
Joined: 17 Apr 2009, 14:21

Postby Koduc » 29 May 2009, 19:00

To kostjn: thx for your staff.

Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?
Koduc
Junior Member
 
Posts: 1
Joined: 27 May 2009, 16:52

Postby kostjn » 30 May 2009, 08:59

Koduc wrote:To kostjn: thx for your staff.

Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?

Yes, certainly.
But now, I am engaged in improvement of a patch for CURRENT.
FreeBSD 8 leaves already soon, therefore I do not see sense to transfer a patch to 7 branch.
kostjn
Junior Member
 
Posts: 7
Joined: 17 Apr 2009, 14:21


Return to FreeBSD Development

Who is online

Users browsing this forum: No registered users and 0 guests