Hi
I have a server:
At a load of 530M symmetric traffic and 81 kpps server loaded completely interrupts:
Configuration:
To balance the queues for processor cores using a script:
To limit the user's bandwidth is used PF + Altq.
For routing using Quagga.
How to fix high interrupt rate?
I have a server:
- Supermicro 6016T-URF
- CPU: 2 x Intel(R) Xeon(R) CPU E5620@2.40GHz (2 x 4 cores, hyper threading disabled)
- RAM: 6Gb (3x2Gb DDR3)
- HDD: Flash 8Gb
- OS: NanoBSD 8.2 p4 amd64
At a load of 530M symmetric traffic and 81 kpps server loaded completely interrupts:
Code:
# top -aSP
last pid: 4616; load averages: 1.91, 1.19, 0.67 up 0+12:49:46 15:44:28
188 processes: 36 running, 118 sleeping, 34 waiting
CPU 0: 0.0% user, 0.0% nice, 2.7% system, 97.3% interrupt, 0.0% idle
CPU 1: 0.0% user, 0.0% nice, 6.0% system, 93.3% interrupt, 0.7% idle
CPU 2: 0.0% user, 0.0% nice, 8.1% system, 91.9% interrupt, 0.0% idle
CPU 3: 0.0% user, 0.0% nice, 2.0% system, 98.0% interrupt, 0.0% idle
CPU 4: 0.0% user, 0.0% nice, 17.4% system, 77.2% interrupt, 5.4% idle
CPU 5: 2.0% user, 0.0% nice, 26.8% system, 63.8% interrupt, 7.4% idle
CPU 6: 0.0% user, 0.0% nice, 8.1% system, 87.2% interrupt, 4.7% idle
CPU 7: 2.7% user, 0.0% nice, 30.9% system, 55.0% interrupt, 11.4% idle
Mem: 38M Active, 13M Inact, 519M Wired, 1128K Cache, 77M Buf, 5338M Free
Swap:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12 root 58 -60 - 0K 928K WAIT 0 21.3H 686.23% [intr]
0 root 40 -68 0 0K 624K - 7 26:11 36.52% [kernel]
11 root 8 171 ki31 0K 128K RUN 0 80.1H 36.04% [idle]
...
Configuration:
Code:
#cat /boot/loader.conf
loader_logo="none"
autoboot_delay="2"
hw.ata.atapi_dma="0"
hw.ata.ata_dma="0"
hw.ata.wc="0"
net.fibs=16
kern.cam.boot_delay=10000
hw.igb.rxd=4096
hw.igb.txd=4096
hw.igb.max_interrupt_rate=1000
Code:
#cat /etc/sysctl.conf
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.drop_redirect=1
net.inet.icmp.log_redirect=1
net.inet.ip.redirect=0
net.inet.tcp.drop_synfin=1
net.inet.ip.forwarding=1
net.inet.flowtable.enable=0
kern.coredump=0
kern.ipc.nmbclusters=512000
dev.igb.0.rx_processing_limit=4096
dev.igb.0.enable_aim=0
dev.igb.0.flow_control=0
dev.igb.1.rx_processing_limit=4096
dev.igb.1.enable_aim=0
dev.igb.1.flow_control=0
dev.igb.2.rx_processing_limit=4096
dev.igb.2.enable_aim=0
dev.igb.2.flow_control=0
dev.igb.3.rx_processing_limit=4096
dev.igb.3.enable_aim=0
dev.igb.3.flow_control=0
Code:
1# cat /etc/start_if.igb0
ifconfig igb0 -rxcsum -txcsum -lro -tso up
ifconfig igb1 -rxcsum -txcsum -lro -tso up
ifconfig igb2 -rxcsum -txcsum -lro -tso up
ifconfig igb3 -rxcsum -txcsum -lro -tso up
....
To balance the queues for processor cores using a script:
Code:
# cat /usr/local/etc/rc.d/cpuset-igb
#!/bin/sh
# PROVIDE: cpuset-igb
# REQUIRE: FILESYSTEMS
# BEFORE: netif
# KEYWORD: nojail
case "$1" in
*start)
echo "Binding igb(4) IRQs to CPUs"
cpus=`sysctl -n kern.smp.cpus`
vmstat -ai | sed -E '/^irq.*que/!d; s/^irq([0-9]+): igb([0-9]+):que ([0-9]+).*/\1 \2 \3/' |\
while read irq igb que
do
cpuset -l $(( ($igb+$que) % $cpus )) -x $irq
done
;;
esac
To limit the user's bandwidth is used PF + Altq.
For routing using Quagga.
How to fix high interrupt rate?