How to get more throughput from/to FreeBSD servers?

[Terri_Kennedy]

We encountered the following:

1. horrible ping times whenever setting MTU > 1500

2. kernel: ix0: Could not setup receive structures

Both of these are due to insufficient buffers, as noted in the driver's README file. Click here and look at lines 271 onward.

sending maxed out at 75 MB/s

Try again after you get the buffer situation sorted out.

 

[yayix]

woohoo!

On iperf server side (receiving end):

 $ netstat -I ix0 -w1 -h

            input          (ix0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         0     0     0          0          0     0          0     0
       19K     0     0       152M       9.5K     0       645K     0
      135K     0     0       1.1G        72K     0       4.8M     0
      138K     0     0       1.2G        74K     0       5.1M     0
      137K     0     0       1.2G        74K     0       5.1M     0
      138K     0     0       1.2G        74K     0       5.1M     0
      137K     0     0       1.2G        74K     0       5.1M     0
      138K     0     0       1.2G        74K     0       5.0M     0
      137K     0     0       1.2G        74K     0       5.0M     0
      137K     0     0       1.2G        74K     0       5.1M     0
      135K     0     0       1.1G        72K     0       4.9M     0
       95K     0     0       825M        51K     0       3.5M     0
         2     0     0       1.8M          0     0       104K     0
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0
         0     0     0          0          0     0          0     0

With just one thread:

# iperf -c 192.168.2.11

------------------------------------------------------------
Client connecting to 192.168.2.11, TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.10 port 57561 connected with 192.168.2.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.90 Gbits/sec

Using a point-to-point 10 Gb connection on PC-BSD installed FreeBSD 9.1-RELEASE and 8.1-RELEASE, and following mods:

/boot/loader.conf

if_ixgbe_load="YES"
kern.ipc.nmbclusters=262144

/etc/sysctl.conf
kern.ipc.nmbjumbop=262144
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216

/etc/rc.conf
ifconfig_ix0="192.168.2.10 netmask 255.255.255.0 mtu 9000"

Driver and slot specifications:
9.1 RELEASE machine

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xec00-0xec1f mem 0xfafe0000-0xfaffffff,0xfaf80000-0xfafbffff,0xfaf7c000-0xfaf7ffff irq 17 at device 0.0 on pci8
ix0: Using MSIX interrupts with 9 vectors
ix0: PCI Express Bus: Speed 2.5Gb/s Width x8

8.1-RELEASE

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.2.0> port 0xcc00-0xcc1f mem 0xfade0000-0xfadfffff,0xfad80000-0xfadbffff,0xfad7c000-0xfad7ffff irq 17 at device 0.0 on pci6
ix0: Using MSIX interrupts with 17 vectors
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: PCI Express Bus: Speed 2.5Gb/s Width x8

Driver version is not that critical for us. What I think made the difference are the tunable and sysctl modifications and making sure ifconfig and route show an MTU of 9000.