Hi.
I am having a problem on processing network traffic on dual processor systems. The machine acts like a filtering 'patch-cord'. Traffic processing is done on netgraph, it's pretty simple:
The general problem is: traffic is not processed on the second CPU and even worse, adding second CPU causes traffic lost and network performance degradation.
On FreeBSD8.2-release all processing (IRQ processes) are done on the first CPU by default, When cpuset is used to bound IRQ processes to the cores of the first CPU, traffic drops disapper.
When cpuset is used to bound any IRQ to the second CPU - system performance degrades, drops appear.
On FreeBSD8.2-stable and 9.0-stable behavior is the same. One CPU works fine, adding a second one causes all cores of the first CPU to be 100% busy with interrupts, and all cores of the second CPU are 100% busy with system load.
When all IRQ processes are bound to the first CPU, the system becomes more responsible, the load of the cores of the first CPU drops to 30-40% (it's proportional to bandwidth), but the system continues to drop traffic.
When the second CPU is physically removed - the system works fine (but its processing capabilities are limited to one CPU).
Some setup details.
CPU's are Xeon5690
NICs are Intel X520-SR2 (E10G42BFSR), ix driver
FreeBSD version is AMD64.
Switching hyperthreading on/off changes number of cores per CPU but makes no difference in terms of overall processing capabilities and behavior.
I would be glad to hear any information and suggestions about the problem.
Thanks.
I am having a problem on processing network traffic on dual processor systems. The machine acts like a filtering 'patch-cord'. Traffic processing is done on netgraph, it's pretty simple:
Code:
ix0:lower<->in:(ng_bpf):match<-->ix1:lowerThe general problem is: traffic is not processed on the second CPU and even worse, adding second CPU causes traffic lost and network performance degradation.
On FreeBSD8.2-release all processing (IRQ processes) are done on the first CPU by default, When cpuset is used to bound IRQ processes to the cores of the first CPU, traffic drops disapper.
When cpuset is used to bound any IRQ to the second CPU - system performance degrades, drops appear.
On FreeBSD8.2-stable and 9.0-stable behavior is the same. One CPU works fine, adding a second one causes all cores of the first CPU to be 100% busy with interrupts, and all cores of the second CPU are 100% busy with system load.
When all IRQ processes are bound to the first CPU, the system becomes more responsible, the load of the cores of the first CPU drops to 30-40% (it's proportional to bandwidth), but the system continues to drop traffic.
When the second CPU is physically removed - the system works fine (but its processing capabilities are limited to one CPU).
Some setup details.
CPU's are Xeon5690
NICs are Intel X520-SR2 (E10G42BFSR), ix driver
FreeBSD version is AMD64.
Switching hyperthreading on/off changes number of cores per CPU but makes no difference in terms of overall processing capabilities and behavior.
I would be glad to hear any information and suggestions about the problem.
Thanks.