MSI/MSI-X on Intel em NIC

Network related discussions (including general TCP/IP stuff, routing, etc).

MSI/MSI-X on Intel em NIC

Postby wrkilu » 15 Nov 2011, 13:02

Hi,

We have a border router with 5 [FILE]em[/FILE] nics. OS is FreeBSD 8.2 x64.
Code: Select all
# top -PSH
last pid:  6097;  load averages:  1.46,  1.45,  1.48                                                   
                                                      up 31+13:32:57  14:02:32
145 processes: 11 running, 111 sleeping, 23 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 30.3% interrupt, 69.7% idle
CPU 1:  0.4% user,  0.0% nice,  0.4% system,  4.5% interrupt, 94.7% idle
CPU 2:  0.0% user,  0.0% nice,  3.8% system, 10.5% interrupt, 85.7% idle
CPU 3:  0.0% user,  0.0% nice, 46.8% system,  0.4% interrupt, 52.8% idle
CPU 4:  0.0% user,  0.0% nice, 29.6% system,  0.0% interrupt, 70.4% idle
CPU 5:  0.4% user,  0.0% nice, 14.7% system,  0.0% interrupt, 85.0% idle
CPU 6:  0.0% user,  0.0% nice, 64.8% system,  0.0% interrupt, 35.2% idle
CPU 7:  0.0% user,  0.0% nice,  0.0% system,  4.9% interrupt, 95.1% idle
Mem: 374M Active, 2576M Inact, 782M Wired, 31M Cache, 412M Buf, 91M Free
Swap: 10G Total, 10G Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root       171 ki31     0K   128K CPU7    7 715.5H 100.00% {idle: cpu7}
   11 root       171 ki31     0K   128K CPU1    1 719.5H 96.53% {idle: cpu1}
   11 root       171 ki31     0K   128K RUN     5 680.7H 86.87% {idle: cpu5}
   11 root       171 ki31     0K   128K CPU2    2 708.4H 78.71% {idle: cpu2}
   11 root       171 ki31     0K   128K CPU4    4 510.5H 72.71% {idle: cpu4}
   11 root       171 ki31     0K   128K CPU0    0 549.6H 71.92% {idle: cpu0}
    0 root       -68    0     0K   208K -       6 380.0H 65.97% {em4 taskq}
   11 root       171 ki31     0K   128K CPU3    3 466.1H 57.86% {idle: cpu3}
    0 root       -68    0     0K   208K CPU3    3 290.3H 45.26% {em1 taskq}
   11 root       171 ki31     0K   128K CPU6    6 384.3H 39.40% {idle: cpu6}
    0 root       -68    0     0K   208K CPU4    4 245.6H 33.84% {em2 taskq}
   12 root       -68    -     0K   368K WAIT    0 204.1H 31.64% {irq256: em0:rx 0}
   12 root       -44    -     0K   368K WAIT    3  81.7H 25.49% {swi1: netisr 0}
    0 root       -68    0     0K   208K -       5  53.9H 14.94% {em3 taskq}
   12 root       -68    -     0K   368K WAIT    1 506:54  1.22% {irq257: em0:tx 0}
    0 root       -68    0     0K   208K -       7  26.5H  0.78% {dummynet}
14434 root        44    0 26984K  5800K select  3 279:09  0.63% snmpd
   12 root       -32    -     0K   368K WAIT    1 243:50  0.00% {swi4: clock}
    0 root       -68    0     0K   208K -       4 139:00  0.00% {em0 txq}
   13 root        44    -     0K    16K -       4  85:22  0.00% yarrow
    0 root       -68    0     0K   208K -       1  64:54  0.00% {em0 rxq}



Hot to turn on MSI on these cards? How to check is it enabled?
wrkilu
Junior Member
 
Posts: 4
Joined: 06 May 2009, 13:06

Postby wblock@ » 15 Nov 2011, 14:52

Check [file]/var/log/messages[/file]:
Code: Select all
em0: Using MSIX interrupts with 3 vectors


I thought there was a sysctl to disable MSI, but looking through the output of [file]sysctl -a[/file] I don't see one for [man=4]em[/man].
User avatar
wblock@
Old Fart
 
Posts: 10169
Joined: 07 Sep 2009, 23:23
Location: Milky Way galaxy

Postby phoenix » 15 Nov 2011, 19:18

It all depends on the version of the [man=4]em[/man] driver, the version of the NIC chipset, and the version of FreeBSD in use. For example, on my 9.0-RC2 box, I have the following sysctl (there's no em-based NIC in this box, though):
Code: Select all
hw.em.enable_msix: 1


On my 8.2-STABLE (r224019) box with this NIC:
Code: Select all
em0@pci0:9:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet

I don't have that sysctl.
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
User avatar
phoenix
MFC'd
 
Posts: 3349
Joined: 17 Nov 2008, 05:43
Location: Kamloops, BC, Canada

Postby wrkilu » 16 Nov 2011, 10:08

Well, we've found the reason, Only em0 on our router supports MSI-X and only she has few interrupts on few cores. In Freebsd 8.2 that feature is enabled in default.

Code: Select all
# pciconv -lcv
em0@pci0:1:0:0: class=0x020000 card=0x34da8086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)
[B]    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled[/B]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected


em1@pci0:2:0:0: class=0x020000 card=0x10828086 chip=0x107d8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 PT'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)


regards
wrkilu
Junior Member
 
Posts: 4
Joined: 06 May 2009, 13:06


Return to Networking

Who is online

Users browsing this forum: No registered users and 1 guest