FreeBSD 8 and if_em. tuning on large trafic

T

tea

Hello All!

Is FreeBSD 8 in the role of shaper.
At 60K packets per second CPU use 70%


CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2671.62-MHz K8-class CPU)

uname -a
Code: 
uname -a
FreeBSD shape.me 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #5: Wed May 19 08:53:50 NOVST 2010     [email]root@shape.me[/email]:/usr/src
/sys/amd64/compile/SHAPER-64  amd64

top -SPH
Code: 
CPU 0:  0.0% user,  0.0% nice, 41.1% system,  8.9% interrupt, 50.0% idle
CPU 1:  0.0% user,  0.0% nice, 41.1% system,  0.0% interrupt, 58.9% idle
Mem: 28M Active, 342M Inact, 232M Wired, 118M Cache, 111M Buf, 264M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME  PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root      171 ki31     0K    32K RUN     0 212:42 64.79% {idle: cpu0}
   11 root      171 ki31     0K    32K CPU1    1 214:58 42.09% {idle: cpu1}
    0 root       54    0     0K   176K CPU1    1  68:18 10.89% {em0_rx_1}
    0 root       54    0     0K   176K WAIT    1  68:13 10.35% {em0_rx_0}
    0 root       52    0     0K   176K WAIT    1  52:43  6.40% {em1_rx_0}
    0 root       53    0     0K   176K WAIT    1  52:41  5.66% {em1_rx_1}
   12 root       16    -     0K   304K WAIT    0  11:52  0.68% {swi16: em0_tx}
   12 root       16    -     0K   304K WAIT    0  11:41  0.68% {swi16: em1_tx}
    0 root      -68    0     0K   176K -       1 116:21  0.00% {dummynet}




i set in loader.conf
Code: 
net.isr.maxthreads=2
net.isr.bindthreads=1

Code: 
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 6.9.14.Yandex[$Revision: 1.36.2.17.2.6 $]
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10b9 subvendor=0x103c subdevice=0x704a class=0x020000
dev.em.0.%parent: pci1
dev.em.0.debug: -1
dev.em.0.stats: -1
dev.em.0.rx_kthreads: 2
dev.em.0.tx_cleanup_threshold: 512
dev.em.0.max0_gprc: 125070
dev.em.0.max0_gptc: 55488
dev.em.0.max0_gorc: 31955593
dev.em.0.max0_gotc: 36289842
dev.em.0.max1_gprc: 125070
dev.em.0.max1_gptc: 55488
dev.em.0.max1_gorc: 31955593
dev.em.0.max1_gotc: 36289842
dev.em.0.max2_gprc: 125070
dev.em.0.max2_gptc: 55488
dev.em.0.max2_gorc: 31955593
dev.em.0.max2_gotc: 36289842
dev.em.0.max3_gprc: 125070
dev.em.0.max3_gptc: 55488
dev.em.0.max3_gorc: 31955593
dev.em.0.max3_gotc: 36289842
dev.em.0.max4_gprc: 125070
dev.em.0.max4_gptc: 55488
dev.em.0.max4_gorc: 31955593
dev.em.0.max4_gotc: 36289842
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 0
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_kthread_priority: 127

Can you please tell what should be done to reduce the configuration burden on the cpu...
And what is the process em0_rx_1 em0_rx_0 ... ?
 
Your top output shows it's 70% idle i.e. not doing anything. Waiting.

Turning on polling may increase the load on your CPU but can increase your throughput.

See tuning(7) for a lot more info.
 
I try turn on polling, but without result. Same effect. :(
Here my screenshots:
network trafic
unlimusagetotday.png

and cpu loading
unlimusagecpusday.png
Click to expand...

Table firewall about 7788 IP addresses, but process dummynet does not load the processor. Or am I wrong? :\

I will not believe that 300 megabits is the limit of network card
peak at 8 am - rebuild kernel

in the evening increases latency network. Emerging packet loss due to high CPU usage. Or is not for him?

what em0_rx_0 ? Packet scheduler?
 
Is polling actually turned on? Just adding it to ifconfig is not enough, it needs to be in the kernel (options DEVICE_POLLING) as well. Only then will polling show up and work):

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=21db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,[B]POLLING[/B],VLAN_HWCSUM,TSO4,WOL_MAGIC>
 
DutchDaemon, polling disabled (remove from kernel configuration). When i'm enable polling, nothing change. CPU (two kernel) is hight, and more network latency.

my interfaces:
Code: 
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:18:71:ec:c7:4d
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:18:71:ec:c5:c8
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 76:eb:97:16:a2:dd
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000

I try to balance interrupts from network cards on two kernel.
Code: 
net.isr.maxthreads=2
net.isr.bindthreads=1

This correctly?
 
Have you confirmed that when you run a kernel with options DEVICE_POLLING and enable it with ifconfig, that it actually appears in the 'options=' list in ifconfig?

If you are finding polling sluggish, you might want to increase kern.hz in /boot/loader.conf to the recommended 2000

Can you actually quantify the network latency you describe?
 
tea said:
Table firewall about 7788 IP addresses, but process dummynet does not load the processor. Or am I wrong? :\
Click to expand...

how many tcp connections are active at the peak?

tea said:
I will not believe that 300 megabits is the limit of network card
Click to expand...

Your mrtg graph shows around 280MBit/s incoming and 170MBit/s outgoing. If iam not wrong we talking about 450 MBit/s total.

I've got the same results between two servers with intel em cards in a 32bit pci slot and with standard cat5 cable. Without polling or any other tuning.
 
tea said:
At 60K packets per second CPU use 70%
Click to expand...
Its all about interrupts and polling must help. Many packets = many interupts (when you work without polling).
Perhaps you forgot `ifconfig em0 polling` after enabling it in kernel?
 
Alt,

From polling in the kernel I refused, because the CPU load increases even more (why??) and there are packet loss. As a workaround I used:
Code: 
sysctl dev.em.1.rx_int_delay = 600

dev.em.1.rx_int_delay: receive interrupt delay in usecs
Click to expand...
 
You must log in or register to reply here.
Back
Top