Packet disorder when routing through FreeBSD box

Darwin · Feb 17, 2015

Hi. I've recently discovered a strange issue I couldn't fix.
The machine terminates like ~1000 vlan interfaces, traffic comes in from igb0 and goes out thru vlans.
So, igb0 is uplink and igb2 is downlink (trunk port for vlans).

Here's the situation:
igb0 receives UDP packets in the right order

Code:

IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:615
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:492
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:615
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:492

and vlan interface transmits them in the wrong order:

Code:

IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:615
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:492
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:492
IP 1.1.1.1.5060 > 2.2.2.2.5060: SIP, length:615

The NIC is Intel 82576, FreeBSD 9.2-STABLE.

loader.conf:

Code:

# cat /boot/loader.conf
net.isr.maxthreads=16
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240

hw.igb.lro=0
hw.igb.rx_process_limit=400
hw.igb.rxd=4096
hw.igb.txd=4096
hw.igb.max_interrupt_rate=32000

kern.ipc.nmbclusters=524288
kern.ipc.nmbjumbop=262144
kern.ipc.maxsockbuf=83886080
hw.intr_storm_threshold=20000

ng_ipfw_load="YES"
alias_ftp_load="YES"
if_igb_load="YES"
geom_mirror_load=YES
ipfw_load="YES"

kern.hz=4000

rc.conf:

Code:

ifconfig_igb0="inet xx.xx.xx.xx/29 up -rxcsum -txcsum -promisc -lro -tso4"
ifconfig_igb2="up -rxcsum -txcsum -lro -promisc -tso4"

No dummynet/filtering rules have been used for _the particular_ host.
I would really be grateful if you could help me solve this issue, thanks.

junovitch@ · Feb 17, 2015

Technically that would be correct. It's TCP that ensures ordering and not UDP. See socket(2).

A SOCK_STREAM type provides sequenced, reliable, two-way connection based byte streams. An out-of-band data transmission mechanism may be supported. A SOCK_DGRAM socket supports datagrams (connectionless, unreliable messages of a fixed (typically small) maximum length).

Have you tried testing without all the tuning? Since you are running multiple netisr(9) threads it could very well be that one packet gets processed faster on one core faster than another packet gets processed on the other core.

Lastly, FreeBSD 9.2 is no longer supported. Please update to FreeBSD 9.3.
https://forums.freebsd.org/threads/topics-about-unsupported-freebsd-versions.40469/

Packet disorder when routing through FreeBSD box

Darwin

junovitch@