1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

MSI/MSI-X on Intel em NIC

Discussion in 'Networking' started by wrkilu, Nov 15, 2011.

  1. wrkilu

    wrkilu New Member

    Messages:
    8
    Thanks Received:
    0
    Hi,

    We have a border router with 5 em nics. OS is FreeBSD 8.2 x64.
    Code:
    # top -PSH
    last pid:  6097;  load averages:  1.46,  1.45,  1.48                                                    
                                                          up 31+13:32:57  14:02:32
    145 processes: 11 running, 111 sleeping, 23 waiting
    CPU 0:  0.0% user,  0.0% nice,  0.0% system, 30.3% interrupt, 69.7% idle
    CPU 1:  0.4% user,  0.0% nice,  0.4% system,  4.5% interrupt, 94.7% idle
    CPU 2:  0.0% user,  0.0% nice,  3.8% system, 10.5% interrupt, 85.7% idle
    CPU 3:  0.0% user,  0.0% nice, 46.8% system,  0.4% interrupt, 52.8% idle
    CPU 4:  0.0% user,  0.0% nice, 29.6% system,  0.0% interrupt, 70.4% idle
    CPU 5:  0.4% user,  0.0% nice, 14.7% system,  0.0% interrupt, 85.0% idle
    CPU 6:  0.0% user,  0.0% nice, 64.8% system,  0.0% interrupt, 35.2% idle
    CPU 7:  0.0% user,  0.0% nice,  0.0% system,  4.9% interrupt, 95.1% idle
    Mem: 374M Active, 2576M Inact, 782M Wired, 31M Cache, 412M Buf, 91M Free
    Swap: 10G Total, 10G Free
    
      PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
       11 root       171 ki31     0K   128K CPU7    7 715.5H 100.00% {idle: cpu7}
       11 root       171 ki31     0K   128K CPU1    1 719.5H 96.53% {idle: cpu1}
       11 root       171 ki31     0K   128K RUN     5 680.7H 86.87% {idle: cpu5}
       11 root       171 ki31     0K   128K CPU2    2 708.4H 78.71% {idle: cpu2}
       11 root       171 ki31     0K   128K CPU4    4 510.5H 72.71% {idle: cpu4}
       11 root       171 ki31     0K   128K CPU0    0 549.6H 71.92% {idle: cpu0}
        0 root       -68    0     0K   208K -       6 380.0H 65.97% {em4 taskq}
       11 root       171 ki31     0K   128K CPU3    3 466.1H 57.86% {idle: cpu3}
        0 root       -68    0     0K   208K CPU3    3 290.3H 45.26% {em1 taskq}
       11 root       171 ki31     0K   128K CPU6    6 384.3H 39.40% {idle: cpu6}
        0 root       -68    0     0K   208K CPU4    4 245.6H 33.84% {em2 taskq}
       12 root       -68    -     0K   368K WAIT    0 204.1H 31.64% {irq256: em0:rx 0}
       12 root       -44    -     0K   368K WAIT    3  81.7H 25.49% {swi1: netisr 0}
        0 root       -68    0     0K   208K -       5  53.9H 14.94% {em3 taskq}
       12 root       -68    -     0K   368K WAIT    1 506:54  1.22% {irq257: em0:tx 0}
        0 root       -68    0     0K   208K -       7  26.5H  0.78% {dummynet}
    14434 root        44    0 26984K  5800K select  3 279:09  0.63% snmpd
       12 root       -32    -     0K   368K WAIT    1 243:50  0.00% {swi4: clock}
        0 root       -68    0     0K   208K -       4 139:00  0.00% {em0 txq}
       13 root        44    -     0K    16K -       4  85:22  0.00% yarrow
        0 root       -68    0     0K   208K -       1  64:54  0.00% {em0 rxq}



    Hot to turn on MSI on these cards? How to check is it enabled?
     
  2. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,526
    Thanks Received:
    2,217
    Check /var/log/messages:
    Code:
    em0: Using MSIX interrupts with 3 vectors
    


    I thought there was a sysctl to disable MSI, but looking through the output of sysctl -a I don't see one for em(4).
     
  3. phoenix

    phoenix Moderator Staff Member Moderator

    Messages:
    3,423
    Thanks Received:
    753
    It all depends on the version of the em(4) driver, the version of the NIC chipset, and the version of FreeBSD in use. For example, on my 9.0-RC2 box, I have the following sysctl (there's no em-based NIC in this box, though):
    Code:
    hw.em.enable_msix: 1


    On my 8.2-STABLE (r224019) box with this NIC:
    Code:
    em0@pci0:9:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = '82571EB Gigabit Ethernet Controller (Copper)'
        class      = network
        subclass   = ethernet

    I don't have that sysctl.
     
  4. wrkilu

    wrkilu New Member

    Messages:
    8
    Thanks Received:
    0
    Well, we've found the reason, Only em0 on our router supports MSI-X and only she has few interrupts on few cores. In Freebsd 8.2 that feature is enabled in default.

    Code:
    # pciconv -lcv
    em0@pci0:1:0:0: class=0x020000 card=0x34da8086 chip=0x10d38086 rev=0x00 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
        class      = network
        subclass   = ethernet
        cap 01[c8] = powerspec 2  supports D0 D3  current D0
        cap 05[d0] = MSI supports 1 message, 64 bit
        cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)
        cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    
    
    em1@pci0:2:0:0: class=0x020000 card=0x10828086 chip=0x107d8086 rev=0x06 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = 'PRO/1000 PT'
        class      = network
        subclass   = ethernet
        cap 01[c8] = powerspec 2  supports D0 D3  current D0
        cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
        cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)


    regards