MSI/MSI-X on Intel em NIC

[wrkilu]

Hi,

We have a border router with 5 em nics. OS is FreeBSD 8.2 x64.

# top -PSH
last pid:  6097;  load averages:  1.46,  1.45,  1.48                                                    
                                                      up 31+13:32:57  14:02:32
145 processes: 11 running, 111 sleeping, 23 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 30.3% interrupt, 69.7% idle
CPU 1:  0.4% user,  0.0% nice,  0.4% system,  4.5% interrupt, 94.7% idle
CPU 2:  0.0% user,  0.0% nice,  3.8% system, 10.5% interrupt, 85.7% idle
CPU 3:  0.0% user,  0.0% nice, 46.8% system,  0.4% interrupt, 52.8% idle
CPU 4:  0.0% user,  0.0% nice, 29.6% system,  0.0% interrupt, 70.4% idle
CPU 5:  0.4% user,  0.0% nice, 14.7% system,  0.0% interrupt, 85.0% idle
CPU 6:  0.0% user,  0.0% nice, 64.8% system,  0.0% interrupt, 35.2% idle
CPU 7:  0.0% user,  0.0% nice,  0.0% system,  4.9% interrupt, 95.1% idle
Mem: 374M Active, 2576M Inact, 782M Wired, 31M Cache, 412M Buf, 91M Free
Swap: 10G Total, 10G Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root       171 ki31     0K   128K CPU7    7 715.5H 100.00% {idle: cpu7}
   11 root       171 ki31     0K   128K CPU1    1 719.5H 96.53% {idle: cpu1}
   11 root       171 ki31     0K   128K RUN     5 680.7H 86.87% {idle: cpu5}
   11 root       171 ki31     0K   128K CPU2    2 708.4H 78.71% {idle: cpu2}
   11 root       171 ki31     0K   128K CPU4    4 510.5H 72.71% {idle: cpu4}
   11 root       171 ki31     0K   128K CPU0    0 549.6H 71.92% {idle: cpu0}
    0 root       -68    0     0K   208K -       6 380.0H 65.97% {em4 taskq}
   11 root       171 ki31     0K   128K CPU3    3 466.1H 57.86% {idle: cpu3}
    0 root       -68    0     0K   208K CPU3    3 290.3H 45.26% {em1 taskq}
   11 root       171 ki31     0K   128K CPU6    6 384.3H 39.40% {idle: cpu6}
    0 root       -68    0     0K   208K CPU4    4 245.6H 33.84% {em2 taskq}
   12 root       -68    -     0K   368K WAIT    0 204.1H 31.64% {irq256: em0:rx 0}
   12 root       -44    -     0K   368K WAIT    3  81.7H 25.49% {swi1: netisr 0}
    0 root       -68    0     0K   208K -       5  53.9H 14.94% {em3 taskq}
   12 root       -68    -     0K   368K WAIT    1 506:54  1.22% {irq257: em0:tx 0}
    0 root       -68    0     0K   208K -       7  26.5H  0.78% {dummynet}
14434 root        44    0 26984K  5800K select  3 279:09  0.63% snmpd
   12 root       -32    -     0K   368K WAIT    1 243:50  0.00% {swi4: clock}
    0 root       -68    0     0K   208K -       4 139:00  0.00% {em0 txq}
   13 root        44    -     0K    16K -       4  85:22  0.00% yarrow
    0 root       -68    0     0K   208K -       1  64:54  0.00% {em0 rxq}

Hot to turn on MSI on these cards? How to check is it enabled?

 

[wblock@]

Check /var/log/messages:

em0: Using MSIX interrupts with 3 vectors

I thought there was a sysctl to disable MSI, but looking through the output of sysctl -a I don't see one for em(4).

 

[phoenix]

It all depends on the version of the em(4) driver, the version of the NIC chipset, and the version of FreeBSD in use. For example, on my 9.0-RC2 box, I have the following sysctl (there's no em-based NIC in this box, though):

hw.em.enable_msix: 1

On my 8.2-STABLE (r224019) box with this NIC:

em0@pci0:9:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet

I don't have that sysctl.

 

[wrkilu]

Well, we've found the reason, Only em0 on our router supports MSI-X and only she has few interrupts on few cores. In Freebsd 8.2 that feature is enabled in default.

# pciconv -lcv
em0@pci0:1:0:0: class=0x020000 card=0x34da8086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)
[B]    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled[/B]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected


em1@pci0:2:0:0: class=0x020000 card=0x10828086 chip=0x107d8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 PT'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)

regards