Huge load spikes when using disk

Pushrod

Pushrod

I have a vps server running FreeBSD 9 amd64. If I do something like install a port, the load goes sky high and the system becomes very sluggish. It recovers just fine though.

Here is some sample top -HS output.

Code: 
last pid: 58642;  load averages:  8.94,  6.70,  4.27
307 processes: 31 running, 263 sleeping, 13 waiting
CPU:  0.0% user,  0.0% nice,  9.7% system, 65.4% interrupt, 24.8% idle
Mem: 491M Active, 261M Inact, 218M Wired, 832K Cache, 110M Buf, 3156K Free
Swap: 4096M Total, 238M Used, 3858M Free, 5% Inuse, 36K Out

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   12 root       -68    -     0K   240K RUN     0 114:51 37.99% intr{swi2: cambio}
   12 root       -88    -     0K   240K WAIT    1  91:40 33.98% intr{irq14: ata0}
   12 root       -60    -     0K   240K WAIT    1 383:24 16.99% intr{swi4: clock}
   13 root        -8    -     0K    48K CPU0    0 299:57 16.99% geom{g_up}
   12 root       -60    -     0K   240K RUN     0   5:07 16.99% intr{swi4: clock}
58466 www         52    0   434M 26120K RUN     0   0:07  9.96% httpd
   11 root       155 ki31     0K    32K RUN     1 386.3H  7.96% idle{idle: cpu1}
   11 root       155 ki31     0K    32K RUN     0 375.4H  6.98% idle{idle: cpu0}
   12 root       -56    -     0K   240K WAIT    0 149:59  6.98% intr{swi5: +}
54851 www         30    0   498M 73204K RUN     0   0:28  3.96% httpd
    5 root       -16    -     0K    16K psleep  1   1:38  1.95% vmdaemon
57436 www         22    0   468M 34864K RUN     0   0:05  1.95% httpd
   16 root       -16    -     0K    16K sdflus  0  84:34  0.98% softdepflush
   14 root       -16    -     0K    16K -       1   1:16  0.98% yarrow
 1422 mysql       42    0  1290M   126M sbwait  0   0:05  0.98% mysqld{mysqld}
56570 www         21    0   492M 32512K biord   0   0:04  0.98% httpd
   12 root       -88    -     0K   240K WAIT    0 212:02  0.00% intr{irq23: uhci0}
   13 root        -8    -     0K    48K RUN     0 148:44  0.00% geom{g_down}
 1422 mysql       20    0  1290M   126M select  1  53:37  0.00% mysqld{mysqld}
 
Can you run # systat -vmstat while the load is high?
 
I just ran these two commands and the load went way up quickly:
[CMD=""]cd /usr/ports/devel/git[/CMD]
[CMD=""]make[/CMD]

Code: 
    2 users    Load  5.11  1.90  1.38                  Aug 22 09:09

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act  517304   25708  7887408    91788   58796  count
All  551804   30808 1081837k   168724          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt    167 cow      55 total
  9         193  21   135  886 1806   18   31  848    512 zfod        atkbd0 1
                                                          ozfod     2 ata0 14
23.8%Sys  70.1%Intr  6.1%User  0.0%Nice  0.0%Idle        %ozfod       ata1 15
|    |    |    |    |    |    |    |    |    |    |       daefr     5 uhci0 23
============+++++++++++++++++++++++++++++++++++>>>    306 prcfr    11 re0 32
                                       345 dtbuf      588 totfr    16 cpu0:timer
Namei     Name-cache   Dir-cache     68577 desvn          react    21 cpu1:timer
   Calls    hits   %    hits   %     23700 numvn          pdwak
    4417    4417 100                 16758 frevn          pdpgs
                                                          intrn
Disks  ada0   cd0 pass0 pass1                      225120 wire
KB/t  44.90  0.00  0.00  0.00                      359428 act
tps       2     0     0     0                      354500 inact
MB/s   0.08  0.00  0.00  0.00                       25964 cache
%busy    35     0     0     0                       32832 free
                                                   112896 buf
 
That look normal.
Maybe SCHED4BSD would be worth a try if you don't mind to compile a kernel.
 
lme@ said:
That look normal.
Maybe SCHED4BSD would be worth a try if you don't mind to compile a kernel.
Click to expand...

I have no issue with compiling a new kernel, but I am concerned that the whole system will be unresponsive during this time. The load spikes can be very, very bad when building software. It goes as high as 17-20 just by untar-ing a large file. It seems there is an interrupt storm with the (fake) ATA controller, but I don't know why.

I am not sure what the host software is, though it may not matter because I can't upgrade it anyways (I am at the mercy of the hosting service).
 
Out of curiosity (and to rule one possible error out) : could it be that the "disc" partition is not 4k aligned, forcing a memcopy of all I/O? (If that is even possible, but maybe this is happening in the fake controller).
 
I also thought first that it was an interrupt storm, but you don't see a lot of interrupts in your output of systat().
 
I made a kernel with SCHED_4BSD and it made no difference in load averages or responsiveness. The difference instead, is the top output shows a lot of cpu going to System instead of Interrupt, and the processes themselves (Apache, Exim, etc) use a whole bunch of cpu for short times.

There is clearly something wrong with FreeBSD 9 amd64 running under this VM software. The performance is unbearably bad.
 
You must log in or register to reply here.
Back
Top