Only two CPUs in use

P

personalraul

Hi all

We are using Dummynet on FreeBSD to emulate networks delay. We are running a high traffic and we have detected some problems. Even with any delay, packet loss etc in the pipes we reach a high use in our cpus.


We have this HW:
Code: 
sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: i386
hw.model: Dual-Core AMD Opteron(tm) Processor 2218
hw.ncpu: 4
hw.machine_arch: i386
When we run our traffic, we can see this info in top :
Code: 
top -aSCHIP

last pid: 13150;  load averages:  0.00,  0.00,  0.00                                                                                                            up 3+17:39:16  10:50:25
80 processes:  7 running, 57 sleeping, 16 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 80.1% interrupt, 19.9% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system, 84.0% interrupt, 16.0% idle
Mem: 14M Active, 145M Inact, 135M Wired, 112M Buf, 2961M Free
Swap: 16G Total, 16G Free

  PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME    CPU COMMAND
   13 root     171 ki31     0K     8K CPU1   1  88.5H 100.00% [idle: cpu1]
   12 root     171 ki31     0K     8K RUN    2  87.7H 100.00% [idle: cpu2]
   38 root     -68    -     0K     8K CPU3   3  98:36 86.47% [irq258: bce0]
   39 root     -68    -     0K     8K CPU0   0  55:41 83.01% [irq259: bce1]
   14 root     171 ki31     0K     8K RUN    0  86.9H 22.90% [idle: cpu0]
   11 root     171 ki31     0K     8K CPU3   3  87.7H 18.95% [idle: cpu3]

top -aSP
last pid: 13170;  load averages:  0.00,  0.00,  0.00                                                                                                            up 3+17:45:06  10:56:15
80 processes:  8 running, 57 sleeping, 15 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 73.8% interrupt, 26.2% idle
CPU 1:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 2:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system, 74.6% interrupt, 25.4% idle
Mem: 14M Active, 145M Inact, 135M Wired, 112M Buf, 2961M Free
Swap: 16G Total, 16G Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   13 root        1 171 ki31     0K     8K RUN    1  88.6H 100.00% [idle: cpu1]
   12 root        1 171 ki31     0K     8K RUN    2  87.8H 100.00% [idle: cpu2]
   38 root        1 -68    -     0K     8K CPU3   3 103:13 83.59% [irq258: bce0]
   39 root        1 -68    -     0K     8K CPU0   0  60:07 79.44% [irq259: bce1]
   14 root        1 171 ki31     0K     8K RUN    0  86.9H 27.00% [idle: cpu0]
   11 root        1 171 ki31     0K     8K RUN    3  87.7H 22.02% [idle: cpu3]

Like you can see, we have 4 CPUs but we are using only 2. An these two cpus (cpu0 and cpu3) have a very high value for interrupt cpu level(84.0% interrupt).

We have tried with these commands, but situation ( only 2 cpu's in use) remains:
Code: 
cpuset -g
pid -1 mask: 0, 1, 2, 3
cpuset -l 0-3 -s 1
cpuset -g
pid -1 mask: 0, 1, 2, 3

Another option is improve system behavior to avoid this high level of interruptions.



Any support will be wellcome:

1- How to force to use 4 cpu's
2- How to improve system behavior to reduce interruptions.

thanks in advances.
 
Hi

Thanks SirDice and phoenix by your answer.

We will try with polling, I will inform you.


About NIC:
Code: 
DUMMYNET06# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:17:08:7e:67:16
        inet 10.42.84.184 netmask 0xffffff00 broadcast 10.42.84.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:17:08:7e:67:17
        media: Ethernet autoselect
        status: no carrier
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:19:bb:2e:57:fe
        inet 10.1.6.2 netmask 0xfffffffc broadcast 10.1.6.3
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:19:bb:2e:57:f2
        inet 10.2.6.2 netmask 0xfffffffc broadcast 10.2.6.3
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:cd:69:e3:81:11
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
DUMMYNET06#
 
AFAIK, bce(4) doesn't support multiple queues. Thus, 1 NIC == 1 queue == 1 CPU used to handle interrupts from that NIC. Since you have 2 NICs, only 2 CPUs will be used.
 
personalraul said:
Could we use it with frebsd FreeBSD 7.2?
Click to expand...
FreeBSD 7.2 is end-of-life since June 2010 and thus unsupported. Update to at least 7.4.
 
Wow, you are running i386 on an Opteron system. Try amd64 -- maybe it will benchmark faster.

What is your output of:
Code: 
sysctl net.isr
Would using using netisr help even though bce only has one queue?
Code: 
cat net.isr.maxthreads=4 >> /boot/loader.conf

You could switch to your em ports, or drop the bce card and use an igb card. Lots of things to test!
 
You must log in or register to reply here.
Back
Top