Intel 10G card issue

[alez]

FreeBSD 11.2-RELEASE
2x10G Intel

ix0@pci0:2:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xde080000, size 524288, enabled
    bar   [18] = type I/O Port, range 32, base 0xe020, size 32, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xde504000, size 16384, enabled
ix1@pci0:2:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xde000000, size 524288, enabled
    bar   [18] = type I/O Port, range 32, base 0xe000, size 32, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xde500000, size 16384, enabled

sysctl.conf

net.inet.ip.fastforwarding=1
net.inet.icmp.icmplim=1000
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
kern.ipc.nmbclusters=262144

kern.ipc.somaxconn=65535

net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=131072

net.raw.recvspace=65535
net.inet.raw.maxdgram=1179648
net.inet.raw.recvspace=1179648

kern.ipc.maxsockbuf=33554432
net.local.stream.sendspace=16776960
net.raw.sendspace=16776960

dev.ix.0.fc=0
dev.ix.1.fc=0

loader.conf

kern.maxusers="1024"

if_em_load="YES"

hw.igb.max_interrupt_rate=32000
net.inet.tcp.tcbhashsize=32000

kern.ipc.nmbclusters=524288
kern.ipc.maxsockbuf=1048576

ix driver <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k>, i also tryed last 3.2.18 driver.

Through the server passes a large UPD traffic, about 700 Mb/s, received through ix0 and sent through ix1. When there is a problem with the connection on ix1, or I manually do ifconfig ix1 down/up, UPD traffic stops leaving ix0, after restarting the software (tried two different software on perl and lua) UPD leaves, but with big losses and I see an error "No buffer space available".

I tried to pass the same traffic through em0, did ifconfig em0 down/up, and on em0 there are no such problems.

What it can be, what is missing and why it happens just after problems with connecting to ix, then after rebooting the server everything works fine and without losses?

In advance, I'm sorry for my English, I use G.T.

 

[Phishfry]

Look up FreeBSD MSI-X interrupts for these adapters.
You will need some sysctl tweaks. Also TSO may cause issues so try toggling that feature.

Here is a real good document regarding the problems. Look at the very bottom. Do not install Intel driver use stock FreeBSD one.
http://manuals.ts.fujitsu.com/files/html/primergy/intel/PROXGB/DOCS/FREEBSD/freebsd.htm

Known Limitations

Under small packets UDP stress test with the 10GbE driver, the FreeBSD system will drop UDP packets, due to socket buffers being full. You may want to change the driver's Flow Control variables to the minimum value for controlling packet reception.

 

[alez]

Thanks, it looks like you're right, right after the server is loaded, I see

# vmstat -i | grep ix1
irq273: ix1:q0                    101600       1888
irq274: ix1:q1                    132476       2462
irq275: ix1:q2                    127252       2365
irq276: ix1:q3                    128252       2384
irq277: ix1:q4                    130860       2432
irq278: ix1:q5                    130485       2425
irq279: ix1:q6                    132158       2456
irq280: ix1:q7                    129302       2403
irq281: ix1:link                       2          0

but after down/up on ix1 I see only the value ix1:q0 is increases.

I set hw.ix.enable_msix=0 in loader.conf, reboot server, and after that the problem disappeared, UPD passes normally after link down/up on ix1, but not immediately, after a while.

But I'm not sure if I did the right thing, even though everything works. And what can you do if you do not disable msix? After all, the problem only appears after problems with connecting to a 10G card

I also read the manual from the link, its copy is in the drivers on the network card. I set hw.intr_storm_threshold=9000, disable TSO , but it did not help at all.