FreeBSD gigabit performance slower then on Linux

paradoxmonkey · Nov 29, 2009

Hi,

I have this damn problem that has been bugging me for a long time now. Running iperf I have found that a linux distro have better gigabit network performance then FreeBSD 7.2. I don't think this the way it suppose to be. Why on Earth would FreeBSDs network performance be slower then Linux on the same machine?

I have already tried with kernel tuning: http://fasterdata.es.net/TCP-tuning/FreeBSD.html

windows box (windows 7, realtek nic) --> server box (FreeBSD, Intel NIC) = < 500 Mbit/s

windows box (windows 7, realtek nic) --> server box (Linux, Intel NIC) = ~ 700 Mbit/s

The two test subjects were the same one, I just changed the server boxs OS to Linux.

The server box is a
Intel Atom @ 1.6Ghz
1GB RAM
Intel Gigabit NIC
Realtek NIC (not used for testing)

dmesg:

Code:

Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.2-RELEASE-p3 #0: Thu Aug 20 20:10:26 UTC 2009
    root@vmbsd72i386:/usr/obj/freenas/usr/src/sys/FREENAS-i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz (1597.60-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x106c2  Stepping = 2
  Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40c39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,xTPR,PDCM,<b22>>
  AMD Features=0x100000<NX>
  AMD Features2=0x1<LAHF>
  Logical CPUs per core: 2
real memory  = 1064304640 (1015 MB)
avail memory = 925745152 (882 MB)
ACPI APIC Table: <INTEL  D945JT  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP/HT): APIC ID:  1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
wlan: mac acl policy registered
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <INTEL D945JT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf140-0xf147 mem 0xdfe80000-0xdfefffff,0xc0000000-0xcfffffff,0xdff00000-0xdff3ffff irq 16 at device 2.0 on pci0
agp0: <Intel 945GME SVGA controller> on vgapci0
agp0: detected 7932k stolen memory
agp0: aperture size is 256M
vgapci1: <VGA-compatible display> mem 0xdfe00000-0xdfe7ffff at device 2.1 on pci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8168B/8168C/8168CP/8168D/8111B/8111C/8111CP PCIe Gigabit Ethernet> port 0xe000-0xe0ff
 mem 0xffd04000-0xffd04fff,0xffd00000-0xffd03fff irq 16 at device 0.0 on pci1
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: Ethernet address: 00:1c:c0:ed:90:d1
re0: [FILTER]
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
ath0: <Atheros 5424/2424> mem 0xdfc00000-0xdfc0ffff irq 17 at device 0.0 on pci2
ath0: [ITHREAD]
ath0: WARNING: using obsoleted if_watchdog interface
ath0: Ethernet address: 00:24:2b:10:6a:cd
ath0: mac 14.2 phy 7.0 radio 10.2
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <PCI bus> on pcib4
uhci0: <UHCI (generic) USB controller> port 0xf080-0xf09f irq 23 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0xf060-0xf07f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0xf040-0xf05f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0xf020-0xf03f irq 16 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xdff41000-0xdff413ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: waiting for BIOS to give up control
usb4: timed out waiting for BIOS
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xd000-0xd03f mem 0xdfb40000-0xdfb5ffff,0xdfb20000-0xdfb3ffff irq 20 at device 0.0 on pci5
em0: [FILTER]
em0: Ethernet address: 00:07:e9:0f:e4:09
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf0f0-0xf0ff irq 18 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
atapci1: <Intel ICH7M SATA300 controller> port 0xf0e0-0xf0e7,0xf0d0-0xf0d3,0xf0c0-0xf0c7,0xf0b0-0xf0b3,0xf0a0-0xf0af mem 0xdff40000-0xdff403ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
speaker0: <PC speaker> port 0x61 on acpi0
cpu0: <ACPI CPU> on acpi0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xcf000-0xcffff,0xd0000-0xd0fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/16 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: <vendor 0x04d9 product 0x1400, class 0/0, rev 1.10/1.43, addr 2> on uhub0
kbd2 at ukbd0
uhid0: <vendor 0x04d9 product 0x1400, class 0/0, rev 1.10/1.43, addr 2> on uhub0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging limited to 5 packets/entry by default
md0: Preloaded image </mfsroot> 98566144 bytes at 0x80ed50b0
GEOM_LABEL: Label for provider md0 is ufsid/4a9431be18a25235.
ad0: 977MB <SanDisk SDCFB-1024 HDX 4.03> at ata0-master PIO4
SMP: AP CPU #1 Launched!
GEOM_LABEL: Label for provider ad0a is ufsid/4a9431db64cccabb.
Trying to mount root from ufs:/dev/md0
GEOM_LABEL: Label ufsid/4a9431be18a25235 removed.
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
GEOM_LABEL: Label for provider md1 is ufsid/4b1342c54923ee27.
GEOM_LABEL: Label ufsid/4b1342c54923ee27 removed.

vermaden · Nov 29, 2009

Try setting these:

Code:

# sysctl kern.random.sys.harvest.ethernet=0
# sysctl kern.random.sys.harvest.point_to_point=0
# sysctl kern.random.sys.harvest.interrupt=0

If that would not help, also try this one:

Code:

# net.isr.direct=0

aragon · Nov 29, 2009

You might want to try FreeBSD 8.0 - its network stack is supposedly better than 7.x.

paradoxmonkey · Nov 30, 2009

vermaden said:
If that would not help, also try this one:

Code:

# net.isr.direct=0

That one worked a bit, I got about 600 mbit/s. It's still not as good as in Linux.

Gone try with FreeBSD 8.0 now.

graudeejs · Nov 30, 2009

tuning(7)

ev-i · Dec 1, 2009

Please, answer the question: what a CPU loading during tests?

none · Dec 6, 2009

well, I think the same ev-i is. I have intel motherboard on work powered by a Core 2 Duo CPU and 7-STABLE from somewhere as of May got me almost 1Gbps:

Code:

[xxx@xxx /root/virtualbox/virtualbox]$ iperf -b 1000M -t 120 -c
10.12.2.211
WARNING: option -b implies udp testing
WARNING: option -b is not valid for server mode
------------------------------------------------------------
Client connecting to 10.12.2.211, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 10.12.1.221 port 64308 connected with 10.12.2.211 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-123.6 sec  13.3 GBytes    923 Mbits/sec

this is FreeBSD amd64 7-STABLE (em nic) to another FreeBSD amd64 8-CURRENT Turion X2 (nfe nic)

you get nothing in logs ?

none

brucec@ · Dec 16, 2009

Have you tried disabling/enabling offloading on the NIC? See the rxcsum, txcsum, tso and lro options in ifconfig(8). I think these can have an impact on performance.