igb & interrupt storm detected

I

ikyb

After upgrade system i have problem with igb driver. Old network interface (em) forward 115-130k pakets on new network adapter igb (Intel 82576) maximum 100-110k packets.

dmesg show
Code: 
...
interrupt storm detected on "irq257:"; throttling interrupt source
...

gw7# vmstat -i
interrupt                          total       rate
irq18: ehci0 uhci5                     2          0
irq19: uhci2 uhci*                192701          2
irq21: uhci1                         765          0
irq23: uhci3 ehci1                     1          0
cpu0: timer                    135336795       1981
irq256: igb0                   190500970       2789
irq257: igb0                    83395261       1221
irq258: igb0                           3          0
cpu5: timer                    135336542       1981
cpu11: timer                   135425550       1983
cpu7: timer                    135336538       1981
cpu15: timer                   135425487       1983
cpu2: timer                    135336605       1981
cpu9: timer                    135425559       1983
cpu1: timer                    135336542       1981
cpu8: timer                    135425558       1983
cpu4: timer                    135336542       1981
cpu10: timer                   135425589       1983
cpu3: timer                    135336617       1981
cpu12: timer                   135425561       1983
cpu6: timer                    135336537       1981
cpu14: timer                   135425527       1983
cpu13: timer                   135425561       1983
Total                         2440186813      35731

gw7# netstat -w1 -i igb0
Code: 
            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
     53902     0   37779755      52207     0   32702384     0
     55127     0   38909372      53402     0   33609417     0
     55576     0   38466648      53549     0   33645440     0

gw7# cat /boot/loader.conf
Code: 
hw.igb.rxd=2048
hw.igb.txd=2048

set dev.igb.0.enable_lro=0 not help

gw7# cat /etc/sysctl.conf
Code: 
vfs.read_max=128
net.inet.ip.fw.one_pass=0
net.inet.ip.fastforwarding=1
net.inet.icmp.icmplim=1000
net.inet.ip.dummynet.io_fast=1


Code: 
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
        ether 00:30:48:ce:fe:6c
        inet xx.yy.16.22 netmask 0xffffffc0 broadcast xx.yy.16.63
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

Old server
2*Xenon E5520 dual port network adapter Intel 82546 (em)
OS Freebsd 8.0-STABLE amd64

New server
2*Xeon X5550 dual port network adapter Intel 82576 (igb)
OS Freebsd 8.0-STABLE amd64

At the time of writing a letter to the load about ~40% of the maximum.

P.S. Server role nat+shape.

What can I do to increase performance ?
 
Are you using natd or pf's nat implementation?

I was told that pf's nat implementation works in the kernel, while natd is a userland application, which means the kernel has to copy each packet in and out of the kernel.

Otherwise, I'm interested in what you come up with, I'll be working with a similar server setup soon.
 
m87carlson said:
Are you using natd or pf's nat implementation?

I was told that pf's nat implementation works in the kernel, while natd is a userland application, which means the kernel has to copy each packet in and out of the kernel.

Otherwise, I'm interested in what you come up with, I'll be working with a similar server setup soon.
Click to expand...

I use ipfw nat (IPFIREWALL_NAT) and DUMMYNET, this design works well (previously used ng_nat). pf nat I have not tried, but I do not think that would be a big difference.

driver downloaded from intel v.1.7.4 leads to the collapse of systems :(

I expect the problem is that the igb driver does not support polling, now ordered a network card on the old chip that supports polling.

P.S. In a network many reports of problems with igb driver but solution is not found ...
 
Try balance over cpus

Execute the follow steps:

cpuset -l 3 -x 256
cpuset -l 4 -x 257
cpuset -l 5 -x 258

Try recompile your kernel with HZ=2000 or above value

Att
Willian Pires Souza
 
You must log in or register to reply here.
Back
Top