NIC interrupt mappings

D

devnull82

Hello Forum,

i test FreeBSD intread of Linux for my loadbalancer since a few days. All was working great. But now i have add some more traffic to it and getting a problem:

I have 2 NICs, one for incomeing traffic and another one for outgoing. The interrupts for each NIC running at only one CPU core. "top -P CC" shows:

Code: 
last pid: 9659; load averages: 1.85, 1.68, 1.69 up 0+20:24:20 12:32:55
248 processes: 2 running, 246 sleeping
CPU 0: 0.0% user, 0.0% nice, 53.3% system, 0.0% interrupt, 46.7% idle
CPU 1: 0.0% user, 0.0% nice, 28.6% system, 0.0% interrupt, 71.4% idle
CPU 2: 0.0% user, 0.0% nice, 7.1% system, 35.7% interrupt, 57.1% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 86.7% interrupt, 13.3% idle
CPU 4: 6.7% user, 0.0% nice, 20.0% system, 0.0% interrupt, 73.3% idle
CPU 5: 0.0% user, 0.0% nice, 40.0% system, 0.0% interrupt, 60.0% idle
CPU 6: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 7: 0.0% user, 0.0% nice, 6.7% system, 0.0% interrupt, 93.3% idle

The interrupts at CPU3, are going up to 100%.

Is it possible to use more cores for one NIC-IRQs? Linux is doing that by default at this hardware.

I can change the irq mapping with "cpuset -l 5 -x 257" to another CPU, but if i use "cpuset -l 0-7 -x 257" the interrupts still running at CPU5.

Greetings
- Michael Schmidt
 
It doesn't show as a supported device in the polling manual, and bce(4) doesn't list the polling option, so it's a no-go.
 
Ok, then i will try it with my 2 onboard intel e1000 NICs, because "em" is listed.
"sysctl -a | grep polling " dont showes anything - So i have to compile the kernel with "options DEVICE_POLLING", right?
 
Yes, it belongs in the kernel, and should then be invoked by ifconfig(8) (just add 'polling' to the ifconfig line in /etc/rc.conf).

It should then show up as:

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=1db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,[B]POLLING[/B],VLAN_HWCSUM,TSO4>

I'm assuming [cmd=]sysctl kern.hz[/cmd] shows 1000? Should be the default on most modern hw.
 
You don't really need polling with em(4) devices. Most support "interrupt moderation" in hardware so they won't fire an interrupt on every packet; interrupts fire after X number of packets, or X number of bytes. (I think there are sysctls to control this on some cards, but have never really checked.)

You can also turn on MSI and MSI-X support via sysctl which configures the cards to use message passing instead of standard interrupts.

And, with em(4) cards, you'll get a taskq thread per NIC that will get scheduled onto separate CPUs.
 
My tests show that already, wirth polling i have cpu.sys usage without polling cpu.interrupts at top. So i disabled it now.

Interrupt moderation / msi-x dont work with the bce by default. Is there a way to enable it?

The bce cards are my favorites, because they have a TCP-Offload-Engine.
 
phoenix said:
You don't really need polling with em(4) devices. Most support "interrupt moderation" in hardware so they won't fire an interrupt on every packet; interrupts fire after X number of packets, or X number of bytes. (I think there are sysctls to control this on some cards, but have never really checked.)
Click to expand...

These probably:

Code: 
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
 
Code: 
dev.bce.0.%desc: HP NC382T PCIe DP Multifunction Gigabit Server Adapter (C0)
dev.bce.0.%driver: bce
dev.bce.0.%location: slot=0 function=0
dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x103c subdevice=0x7059 class=0x020000
dev.bce.0.%parent: pci3
dev.bce.0.l2fhdr_error_count: 0
dev.bce.0.mbuf_alloc_failed_count: 0
dev.bce.0.fragmented_mbuf_count: 0
dev.bce.0.dma_map_addr_rx_failed_count: 0
dev.bce.0.dma_map_addr_tx_failed_count: 0
dev.bce.0.unexpected_attention_count: 0
dev.bce.0.stat_IfHcInOctets: 1282458072
dev.bce.0.stat_IfHCInBadOctets: 8674506
dev.bce.0.stat_IfHCOutOctets: 530572072
dev.bce.0.stat_IfHCOutBadOctets: 0
dev.bce.0.stat_IfHCInUcastPkts: 2407689
dev.bce.0.stat_IfHCInMulticastPkts: 0
dev.bce.0.stat_IfHCInBroadcastPkts: 19124
dev.bce.0.stat_IfHCOutUcastPkts: 2337928
dev.bce.0.stat_IfHCOutMulticastPkts: 710
dev.bce.0.stat_IfHCOutBroadcastPkts: 427
dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.0.stat_Dot3StatsFCSErrors: 0
dev.bce.0.stat_Dot3StatsAlignmentErrors: 0
dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.0.stat_Dot3StatsLateCollisions: 0
dev.bce.0.stat_EtherStatsCollisions: 0
dev.bce.0.stat_EtherStatsFragments: 0
dev.bce.0.stat_EtherStatsJabbers: 0
dev.bce.0.stat_EtherStatsUndersizePkts: 0
dev.bce.0.stat_EtherStatsOversizePkts: 0
dev.bce.0.stat_EtherStatsPktsRx64Octets: 27689
dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 1413849
dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 14433
dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 179619
dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 90819
dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 700404
dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.0.stat_EtherStatsPktsTx64Octets: 11130
dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 1964880
dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 6022
dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 18122
dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 66904
dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 272007
dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.0.stat_XonPauseFramesReceived: 0
dev.bce.0.stat_XoffPauseFramesReceived: 0
dev.bce.0.stat_OutXonSent: 0
dev.bce.0.stat_OutXoffSent: 0
dev.bce.0.stat_FlowControlDone: 0
dev.bce.0.stat_MacControlFramesReceived: 0
dev.bce.0.stat_XoffStateEntered: 0
dev.bce.0.stat_IfInFramesL2FilterDiscards: 117167
dev.bce.0.stat_IfInRuleCheckerDiscards: 0
dev.bce.0.stat_IfInFTQDiscards: 0
dev.bce.0.stat_IfInMBUFDiscards: 0
dev.bce.0.stat_IfInRuleCheckerP4Hit: 0
dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.0.stat_CatchupInFTQDiscards: 0
dev.bce.0.stat_CatchupInMBUFDiscards: 0
dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.0.com_no_buffers: 41
dev.bce.1.%desc: HP NC382T PCIe DP Multifunction Gigabit Server Adapter (C0)
dev.bce.1.%driver: bce
dev.bce.1.%location: slot=0 function=1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x103c subdevice=0x7059 class=0x020000
dev.bce.1.%parent: pci3
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.mbuf_alloc_failed_count: 0
dev.bce.1.fragmented_mbuf_count: 0
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 0
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.stat_IfHcInOctets: 0
dev.bce.1.stat_IfHCInBadOctets: 0
dev.bce.1.stat_IfHCOutOctets: 0
dev.bce.1.stat_IfHCOutBadOctets: 0
dev.bce.1.stat_IfHCInUcastPkts: 0
dev.bce.1.stat_IfHCInMulticastPkts: 0
dev.bce.1.stat_IfHCInBroadcastPkts: 0
dev.bce.1.stat_IfHCOutUcastPkts: 0
dev.bce.1.stat_IfHCOutMulticastPkts: 0
dev.bce.1.stat_IfHCOutBroadcastPkts: 0
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_EtherStatsCollisions: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsPktsRx64Octets: 0
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 0
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 0
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 0
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 0
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 0
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx64Octets: 0
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 0
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 0
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 0
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 0
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 0
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_XonPauseFramesReceived: 0
dev.bce.1.stat_XoffPauseFramesReceived: 0
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 0
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 0
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
 
devnull82 said:
My tests show that already, wirth polling i have cpu.sys usage without polling cpu.interrupts at top. So i disabled it now.

Interrupt moderation / msi-x dont work with the bce by default. Is there a way to enable it?

The bce cards are my favorites, because they have a TCP-Offload-Engine.
Click to expand...

Does TOE appear in the ifconfig output for bce devices? Are you sure the FreeBSD driver for bce supports TOE? Looking through if_bce.c, I don't see any mention of TOE, just TSO4 (TCP Segment Offload, I think). em(4) supports TSO4 as well.

Going through the sources for 7.2, the only driver that supports TOE is the Chelsio 10 GBE driver cxgb(4). Same with the sources for 8.0. There's work ongoing to add hooks for TOE into the network stack, but nothing has been committed yet.

Just curious, as I've never had any good experiences with any kind of Broadcom networking gear, and have avoided it like the plague for the last couple of years.

Looking into it further, it seems Linux doesn't have any TOE support either.
 
devnull82 said:
Interrupt moderation / msi-x dont work with the bce by default. Is there a way to enable it?
Click to expand...

These are what I have:
Code: 
hw.pci.honor_msi_blacklist: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1

I haven't set them, so they're the defaults.
 
You must log in or register to reply here.
Back
Top