(sorry for cross-posting this here and to freebsd-net@ in case they have different audiences)
Hi, we're a medium sized ISP that need to pass all incoming user traffic through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until yesterday it had two 1 gb em cards, one for input, one for output. As we were approaching the bandwidth limitation we switched the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough opportunity to test the newer FreeBSD under load, but from the time we did and I know, the same slowness started happening at about 300-400 mbps load. There are no more fragmentation failed errors. No evident drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 and ix1 are configured at their default settings.
Then I read something about the number of ixgbe device descriptors (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to 4096 permittable. But after some grepping on the source tree I saw that contrary to what the old docs say they are both set to an optimal value:
/sys/dev/ixgbe/ixgbe.c:
So, here's my kernel config for your viewing pleasure:
Here's /etc/sysctl.conf:
Any tips? I'll be happy to try and add some more info upon request.
Thanks.
Hi, we're a medium sized ISP that need to pass all incoming user traffic through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until yesterday it had two 1 gb em cards, one for input, one for output. As we were approaching the bandwidth limitation we switched the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough opportunity to test the newer FreeBSD under load, but from the time we did and I know, the same slowness started happening at about 300-400 mbps load. There are no more fragmentation failed errors. No evident drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 and ix1 are configured at their default settings.
Then I read something about the number of ixgbe device descriptors (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to 4096 permittable. But after some grepping on the source tree I saw that contrary to what the old docs say they are both set to an optimal value:
/sys/dev/ixgbe/ixgbe.c:
Code:
/*
** Number of TX descriptors per ring,
** setting higher than RX as this seems
** the better performing choice.
*/
static int ixgbe_txd = PERFORM_TXD;
TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);
/* Number of RX descriptors per ring */
static int ixgbe_rxd = PERFORM_RXD;
TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
Code:
/sys/dev/ixgbe/ixgbe.h:
/*
* TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
* number of transmit descriptors allocated by the driver. Increasing this
* value allows the driver to queue more transmits. Each descriptor is 16
* bytes. Performance tests have show the 2K value to be optimal for top
* performance.
*/
#define DEFAULT_TXD 1024
#define PERFORM_TXD 2048
#define MAX_TXD 4096
#define MIN_TXD 64So, here's my kernel config for your viewing pleasure:
Code:
include GENERIC
ident SHAPER
nomakeoptions DEBUG
nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4
nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5
nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
nooptions COMPAT_FREEBSD32 # Compatible with i386 binaries
nooptions INET6 # IPv6 communications protocols
options ZERO_COPY_SOCKETS
# XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work
#options DEVICE_POLLING
options IPFIREWALL #firewall
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by defaultHere's /etc/sysctl.conf:
Code:
net.inet.ip.fw.verbose=0
kern.ipc.shmall=65536
kern.ipc.shmmax=268435456
kern.ipc.semmap=1024
kern.ipc.nmbclusters=111111
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3??
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000
hw.intr_storm_threshold=9000
#dev.em.0.rx_processing_limit=-1 # device not used any moreAny tips? I'll be happy to try and add some more info upon request.
Thanks.
The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf:
Thanks for your cooperation.