Testing FreeBSD 10.0 - strange values in top

Hi,

Today I've installed a 10.0 FreeBSD in our VMware environment, just checking things out. I've chosen an install on ZFS (one disk, so 'striping'), with encryption. The installation went fine and the machine boots fine. When I login, and run top, this puzzles me:

Code:
last pid:  1143;  load averages:  0.20,  0.18,  0.19    up 0+00:18:59  13:33:36
18 processes:  1 running, 17 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 14M Active, 14M Inact, 61M Wired, 884M Free
ARC: 23M Total, 6974K MFU, 14M MRU, 32K Anon, 262K Header, 1204K Other
Swap: 1024M Total, 1024M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
 1137 root          1  20    0 19768K  2660K RUN      0:00   0.00% top
 1114 root          1  20    0 86084K  6896K select   0:00   0.00% sshd
 1117 root          1  20    0 23492K  3596K pause    0:00   0.00% csh
 1057 root          1  20    0 23980K  5332K select   0:00   0.00% sendmail
 1101 root          1  24    0 47656K  2600K wait     0:00   0.00% login
 1109 root          1  23    0 23492K  3224K ttyin    0:00   0.00% csh
  899 root          1  20    0 14424K  1976K select   0:00   0.00% syslogd
 1064 root          1  20    0 16520K  2076K nanslp   0:00   0.00% cron
 1102 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1108 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1029 root          1  20    0 60816K  6076K select   0:00   0.00% sshd
 1106 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1103 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1105 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1104 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
 1107 root          1  52    0 14420K  1880K ttyin    0:00   0.00% getty
  780 root          1  20    0 13584K  4532K select   0:00   0.00% devd
 1060 smmsp         1  52    0 23980K  4968K pause    0:00   0.00% sendmail

So, it's 100% idle. Nothing really 'running', but still the load average values are far from zero. I've seen this before in old versions like 6.x when running in VMware, but I haven't seen it since the last 9.x (9.2) version of FreeBSD, until now. Should I worry about this?
 
Load averages are also depending on the number of processes that are running, even if they don't do anything.
 
To expand on @SirDice's comment:
  • A load average of 1 means there is one process waiting for something (usually the CPU or disk).
  • A load average of 2 means there are two processes waiting.
  • A load average of 0.5 (over the last minute), can mean that for 30 seconds, there was one process waiting, and for 30 seconds, there were no processes waiting. It can also mean that for 15 seconds there were two processes waiting, and for 45 seconds there were no processes waiting. The keyword here is average.
A process will always do something occasionally, even an 'idle' process will use up a (very small) amount of system activity. This is why a very large amount of processes results in a fairly high load average.

That being said, it's not clear if this is the actual case. The astute reader will have noticed that the load average increases when processes are waiting, this may be waiting for the CPU, but may also be for something else (disk I/O, network).

You can use the m key to switch to top's I/O mode, which may or may not show something interesting.
 
Last edited by a moderator:
Thanks for that explanation, those numbers have always been quite unclear to me. But I did know you can't really tell much from just that number, with a load average of 2.0 you still don't quite know if the machine is really busy or there's just a load of processes. To be honest I never look at them. Percentages "busy" for I/O and CPU are much more interesting.
 
SirDice said:
Load averages are also depending on the number of processes that are running, even if they don't do anything.

Eh .. how come I do not see this on ANY other FreeBSD server that's doing nothing? (only tried in Vbox and Vmware so far, not on physical yet)
 
Ok, so here's some more data that I can't explain. This is as a VM in ESXi5.1 (enterprise license), and booted into single user mode. Started top -aS -s10, and left it for a while:

freebsd10-vmware.png


load average of 0.5? What?

Dmesg from this VM:

Code:
FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
    root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz (2266.75-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206c2  Family = 0x6  Model = 0x2c  Stepping = 2
  Features=0xfa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1073741824 (1024 MB)
avail memory = 1009299456 (962 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <PTLTD          APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
acpi0: <INTEL 440BX> on motherboard
acpi0: Power Button (fixed)
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0

Later on I installed it on a physical machine. Booted into single user mode, top -Sa -s10 for a while, then a screenshot:

freebsd10-physical.png


Less serious here, but still a load of almost 0.3.

Dmesg from this physical machine:

Code:
FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
    root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz (3100.10-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306a9  Family = 0x6  Model = 0x3a  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
  TSC: P-state invariant, performance statistics
real memory  = 8598323200 (8200 MB)
avail memory = 8244502528 (7862 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
random: <Software, Yarrow> initialized
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
 
Load averages are not static numbers that mean anything unto themselves. Instead, they should be considered relative to what's normal for that system.

IOW, watch the load average for a few minutes after boot and after things have settled down to "normal" for that system. That load average (even if it's 15.0) is the baseline. Any jumps above that number are "bad".

Having a high load average (and nothing under 2.0 can be considered "high") is not bad by itself, so long as the system is still responsive and does what's its supposed to.

IOW, you're worrying about nothing.
 
phoenix said:
Load averages are not static numbers that mean anything unto themselves. Instead, they should be considered relative to what's normal for that system.

IOW, watch the load average for a few minutes after boot and after things have settled down to "normal" for that system. That load average (even if it's 15.0) is the baseline. Any jumps above that number are "bad".

Having a high load average (and nothing under 2.0 can be considered "high") is not bad by itself, so long as the system is still responsive and does what's its supposed to.

IOW, you're worrying about nothing.

So what you are saying is that since FreeBSD 10.0, we have load averages of !0 when a machine is doing nothing (even 10+ minutes idle single user mode). Some random number will eventually be the baseline and than that's our reference?

Right!
 
frijsdijk said:
I've chosen an install on ZFS (one disk, so 'striping'), with encryption.

Off-topic, perhaps, but how was the performance with this ZFS setup? Would there be any benefit to using multiple virtual disks to create a raidz pool?
 
frijsdijk said:
So what you are saying is that since FreeBSD 10.0, we have load averages of !0 when a machine is doing nothing (even 10+ minutes idle single user mode).
The OS is never doing nothing. Especially if there are a lot of processes.
 
I have strange indication with top to but in the user column ... the 88 for Mysql and <sh>

I haven't any "88" user ...

Code:
last pid: 39501;  load averages:  0.32,  0.30,  0.26   up 0+14:45:48  13:26:49
44 processes:  1 running, 43 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 58M Active, 69M Inact, 1301M Wired, 6456K Cache, 408M Free
ARC: 768M Total, 261M MFU, 375M MRU, 16K Anon, 46M Header, 86M Other
Swap: 4096M Total, 57M Used, 4039M Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1741     88       19  35    0   285M 26892K sigwai  1   0:49   0.00% mysqld
17019 www           1  28    0   215M 17248K accept  0   0:05   0.00% php-fpm
17018 www           1  20    0   215M  9752K accept  0   0:02   0.00% php-fpm
  686 _pflogd       1  20    0 14600K   896K bpf     0   0:02   0.00% pflogd
 1807 root          1  20    0 23980K  2244K select  0   0:01   0.00% sendmail
17162 www           1  35   15 38680K  1584K kqread  1   0:00   0.00% nginx
17017 root          1  20    0   211M  5172K kqread  1   0:00   0.00% php-fpm
  868 root          1  20    0 14424K  1096K select  1   0:00   0.00% syslogd
39399 mathieu       1  20    0 86084K  5596K select  1   0:00   0.00% sshd
 1765 root          1  20    0 16520K   408K nanslp  0   0:00   0.00% cron
 1814 root          1  20    0 16520K   544K nanslp  1   0:00   0.00% cron
 1292 root          1  20    0 14432K   776K select  1   0:00   0.00% syslogd
39421 root          1  20    0 23492K  3160K pause   1   0:00   0.00% csh
 1351     88        1  52    0 16988K     0K wait    0   0:00   0.00% <sh>
 
oxyaxion said:
I haven't any "88" user ...
You should have.

Code:
root@mysql:~ # id 88
uid=88(mysql) gid=88(mysql) groups=88(mysql)
root@mysql:~ # getent passwd mysql
mysql:*:88:88:MySQL Daemon:/var/db/mysql:/usr/sbin/nologin
 
Back
Top