1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

CPU limit for jails under ULE scheduler

Discussion in 'FreeBSD Development' started by kostjn, Apr 17, 2009.

  1. kostjn

    kostjn New Member

    Messages:
    7
    Thanks Received:
    1
    Hello all!
    I`m rewrire original cdjones patch for cpu limit for jail under ULE scheduler.
    So, this work simple.
    We count cpu usage for all jails, and if jail use cpu more than have shared cpu, we move his threads to IDLE queue.
    Jailed thread can use all avaliable cpu time, if system has avaliable cpu.
    If system under heavy load, jailed thread can`t use cpu long as ratio (shared cpu for jail/ all shared cpu) < (estimate usage cpu for jail / all usage cpu) .
    Unjailed thread are not subject to this regime.
    Interactive thread also are not subject to this regime.
    Add 2 sysctl
    kern.sched.total_sched_shares - total count shares cpu in system, increase if we have more cpu
    kern.sched.flush_estcpu_interval - flush estcpu interval in ticks, default is 2560 = 2 * 128 * 10, NCPU*stathz*sec, increase if we have more cpu
    For use cpu limit, you need use flag -S NSharedCPU in /usr/sbin/jail program.
    My example
    Code:
    jail -S100 /usr/jails/root/ root.kostjn.pht  192.168.0.245 /bin/csh


    I`m tested this under 10 simultaneous process in jail and in main system. test program is infinity cycle an 8 core xeon, use RELENG_7.
    First run process in jail, and after in main system.
    This one process tracking cpu usage
    Jail
    Code:
    root    1052  0.0  0.0  3692   784  p1  RJ    7:38PM   0:00.39 /test.o
    root    1052 21.2  0.0  3692   784  p1  RJ    7:38PM   0:02.40 /test.o
    root    1052 35.6  0.0  3692   784  p1  RJ    7:38PM   0:04.40 /test.o
    root    1052 47.5  0.0  3692   784  p1  RJ    7:38PM   0:06.41 /test.o
    root    1052 39.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 33.2  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 27.6  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 22.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 19.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 15.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 13.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052 10.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
    root    1052  8.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /tes
    

    Main system
    Code:
    root    1088 14.9  0.0  3692   780  p0  R     7:38PM   0:01.57 /root/test.o
    root    1088 30.8  0.0  3692   780  p0  R     7:38PM   0:03.60 /root/test.o
    root    1088 43.8  0.0  3692   780  p0  R     7:38PM   0:05.60 /root/test.o
    root    1088 51.0  0.0  3692   780  p0  R     7:38PM   0:07.25 /root/test.o
    root    1088 50.8  0.0  3692   780  p0  R     7:38PM   0:08.28 /root/test.o
    root    1088 49.1  0.0  3692   780  p0  R     7:38PM   0:09.21 /root/test.o
    root    1088 48.1  0.0  3692   780  p0  R     7:38PM   0:10.24 /root/test.o
    root    1088 46.2  0.0  3692   780  p0  R     7:38PM   0:11.17 /root/test.o
    root    1088 42.9  0.0  3692   780  p0  R     7:38PM   0:11.95 /root/test.o
    

    So we see, that after run in main system, jailed process can`t usage cpu.

    I`m don`t have big expirience in kernel programming, consequently best if you see source code.
    Please test this patch and post this you restult or all problem.
    This is initial version, without tune jail parameter in runtime.
    Thank.
    Original cdjones cpu and memory limit patch http://wiki.freebsd.org/JailResourceLimits
     
  2. kostjn

    kostjn New Member

    Messages:
    7
    Thanks Received:
    1
  3. DutchDaemon

    DutchDaemon Administrator Staff Member Administrator Moderator

    Messages:
    10,730
    Thanks Received:
    2,133
  4. kostjn

    kostjn New Member

    Messages:
    7
    Thanks Received:
    1
    Jail limit for CURRENT

    Hi.
    I`m rewrite jail limit patch under CURRENT.
    New patch limited CPU, memory, filedesc, process.
    And allow change limit on the fly.

    ===========================================================================
    How to use.
    ===========================================================================
    Build
    Code:
    cvsup CURRENT
    cd /usr/src
    patch -p0 < patch-jail-limit-8CURRENT
    make buildkernel
    make buildworld
    make installkernel
    reboot
    make installworld


    Create new entry in login.conf, for example class jail128
    Code:
    jail128:\
    	:cputime=10:\
    	:memoryuse=128M:\
    	:maxproc=256:\
    	:openfiles=1024:\
    	:tc=default:

    Cputime is percent on 1 core.
    Openfiles is sum filedesc for all proc in jail.

    Create new jail.
    ...
    Add in /etc/rc.conf
    Code:
    	jail_test_flags="-Ljail128"


    Run new jail
    Code:
    /etc/rc.d/jail start test


    ===========================================================================
    Sysctl
    ===========================================================================
    Code:
    [root@book ~]# sysctl security.jail.limit
    security.jail.limit.enable: 1
    security.jail.limit.memory_exceed_kill: 0
    [root@book ~]# sysctl -d security.jail.limit
    security.jail.limit: Jail limit
    security.jail.limit.enable: Enable jail limit
    security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee
    d memory limit



    ===========================================================================
    Jset and Jget
    ===========================================================================
    jset and jget is program for set new jail limit and get current limit
    Example
    Code:
    [root@book ~]# cat /etc/rc.conf | grep jail2
    jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10"
    jail_jail2_rootdir="/usr/jails/jail2/"
    jail_jail2_hostname="jail2.book.pht"
    jail_jail2_interface="re0"
    jail_jail2_ip="192.168.200.22"
    jail_jail2_flags="-Ljail64"
    [root@book ~]# /etc/rc.d/jail start jail2
    Configuring jails:.
    Starting jails: jail2.book.pht.
    [root@book ~]# cd ~kostjn/
    
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
    
    [root@book /home/kostjn]# ./jset.o 1 jail2048
    Set new jail limits, jid = 1
    Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
    [root@book /home/kostjn]# ./jget.o 1
    
    Jail limits and rusage, jid = 1
    Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
    Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65


    You see that new limit is set.

    ===========================================================================
    Test
    ===========================================================================
    Cpu limit
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    Script
    Code:
    [root@book /home/kostjn]# cat test.sh
    #!/bin/sh
    for i in `jot 8 1`; do cpuset -l0  jexec 1 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 2 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 3 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 4 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 5 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 6 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 7 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 8 /a.out & done
    for i in `jot 8 1`; do cpuset -l0  jexec 9 /a.out & done
    
    cpuset -l0  jexec 10 /a.out &
    
    Set class for all jail.
    
    [root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done
    Set new jail limits, jid = 1
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 2
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 3
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 4
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 5
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 6
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 7
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 8
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 9
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    Set new jail limits, jid = 10
    Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
    
    [root@book /home/kostjn]# jexec 1 bash
    [root@jail1 /]# cat cpu.c
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    #include <math.h>
    
    int main(int argc,char *argv[]){
            int64_t i,j=0;
            char *s;
            for (;;){
            }
    
    }
    
    Run test.sh
    
    Result 
    top
    last pid:  3513;  load averages: 70.87, 37.58, 16.40    up 0+00:44:02  14:19:46
    185 processes: 74 running, 111 sleeping
    CPU: 49.9% user,  0.0% nice,  0.0% system,  0.2% interrupt, 49.9% idle
    Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free
    Swap: 4044M Total, 4044M Free
    
      PID JID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAN
     3502  10 root          1  97    0  1480K  1244K CPU0    0   0:13  8.79% a.out
     3474   6 root          1  97    0  1480K  1244K RUN     0   0:04  4.69% a.out
     3431   2 root          1  96    0  1480K  1244K RUN     0   0:03  4.30% a.out
     3454   4 root          1  97    0  1480K  1244K RUN     0   0:03  4.05% a.out
     3422   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.86% a.out
     3482   7 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
     3447   3 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
     3429   1 root          1  96    0  1480K  1244K RUN     0   0:03  3.66% a.out
     3485   8 root          1  97    0  1480K  1244K RUN     0   0:05  3.56% a.out
     3424   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.56% a.out
     3464   5 root          1  97    0  1480K  1244K RUN     0   0:02  3.56% a.out
     3438   2 root          1  96    0  1480K  1244K RUN     0   0:03  3.47% a.out
     3494   9 root          1  96    0  1480K  1244K RUN     0   0:03  3.27% a.out
     3497   9 root          1  97    0  1480K  1244K RUN     0   0:05  3.17% a.out
     3433   2 root          1  96    0  1480K  1244K RUN     0   0:03  2.88% a.out
     3428   1 root          1  96    0  1480K  1244K RUN     0   0:02  2.88% a.out
     3487   8 root          1  97    0  1480K  1244K RUN     0   0:04  2.78% a.out
    


    We see that jail 10 (1 thread), used ~10 % cpu under heavy load.
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    Resourse compute
    Code:
    [root@book /home/kostjn]# ./jset.o 1 jail64
    Set new jail limits, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
    [root@book /home/kostjn]#
    [root@book /home/kostjn]# jexec 1 bash
    [root@jail1 /]# apachectl stop
    /usr/local/sbin/apachectl stop: httpd stopped
    [root@jail1 /]# exit
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24


    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    Resource limit
    Code:
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
    [root@book /home/kostjn]# jexec 1 bash
    [root@jail1 /]# cat mem.c
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    #include <math.h>
    
    int main(int argc,char *argv[]){
            int64_t i,j=0;
            char *s;
            for (i=0; i  < 1000 ;i++){
                    s = malloc(100000 * sizeof(char));
            }
            sleep(1000);
    }
    
    [root@jail1 /]# cc mem.c && ./a.out &
    [1] 1320
    [root@jail1 /]# ls
    bash: fork: Cannot allocate memory
    [root@jail1 /]# exit
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31


    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    We see that jail exceed memory limit. And new fork, mmap syscall not
    permitted.
    If you set sysctl

    Code:
    [root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1
    security.jail.limit.memory_exceed_kill: 1 -> 1
    [root@book /home/kostjn]# ./jget.o 1
    Jail limits and rusage, jid = 1
    Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
    Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
    [root@book /home/kostjn]# jexec 1 bash
    [root@jail1 /]# ./a.out
    Killed: 9
    [root@jail1 /]# exit
    [root@book /home/kostjn]# tail -n 1 /var/log/messages
    May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e
    xceed memory limit


    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    If you attempt set nonexisten class, limit set to infinity.

    Code:
    [root@book /home/kostjn]# ./jset.o 1 jail123
    Set new jail limits, jid = 1
    Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223
    372036854775807



    ===========================================================================
    Problem
    ===========================================================================
    If you have problem in this patch.
    Add to kernel config
    Code:
    options     KTR
    options     KTR_ENTRIES=1024
    options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N)
    options     KTR_MASK=KTR_JAIL
    options     KTR_CPUMASK=0x3
    options     KTR_VERBOSE
    options     PRINTF_BUFR_SIZE=128
    

    Rebuild kernel.
    Reboot.
    Set sysctl
    Code:
    sysctl debug.ktr.mask=65536

    and check /var/log/messages
     
    Koduc thanks for this.
  5. vivek

    vivek New Member

    Messages:
    809
    Thanks Received:
    184
    Thanks for posting this. Have you tested it with 72RELEASE?
     
  6. kostjn

    kostjn New Member

    Messages:
    7
    Thanks Received:
    1
    This patch will not work in 7-STABLE.
    The reason in that in CURRENT developers have altered a code jail, new system calls and so on have been added.
     
  7. Koduc

    Koduc New Member

    Messages:
    1
    Thanks Received:
    0
    To kostjn: thx for your staff.

    Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?
     
  8. kostjn

    kostjn New Member

    Messages:
    7
    Thanks Received:
    1
    Yes, certainly.
    But now, I am engaged in improvement of a patch for CURRENT.
    FreeBSD 8 leaves already soon, therefore I do not see sense to transfer a patch to 7 branch.