Slow upload (IPFW + NATD + amd64) [Solved with PF]

Hello,

My server was working well. I have bought a new 64 Bit Quad server. My old server were 32 bit and working very well with the same configuration. I have installed FreeBSD 8.0-p3 amd64 and setup the system as how I did before. I'm using the same configuration on 3 gateway server and they are all working well.

With this new server, while divert natd is defined in ipfw rules, the upload speed is between 10-20 kb/s

If you remove the natd line, it's transferring with the all capacity without problem.

Code:
server# ipfw list
00010 allow ip from me to me
00100 divert 8668 ip from any to any via fxp0 # if i remove this rule and disable natd, it's transferring normally.
00500 allow ip from any to any
65535 deny ip from any to any

Before digging the configurations, I have changed ethernet nics, cables and all.

Because of this configuration works well with my all machines which are i386 freebsd, I think that something may be different in amd64 systems. Of course, I maybe wrong.

I'm waiting for you suggestions. I will go on a business trip in 2 hours. So I can't reply you in 3-4 days. I will read and reply your thoughts when I returned from trip.

Thank you all.
 
Hello, I'm back.

I want to describe the status a little more.

There is a i386 gateway which connects servers to the internet. Each server is connected to an ethernet nic.

The problem server is a new amd64. It was working very well with same config on i386 machine. I have not changed anything in configuration.

Nat, ipfw etc. are configured in loader.conf and rc.conf. I'm not compiling kernel.

This machine has vlan for jails. It's own ipfw and nat for forwarding ports to jails. When this machine uploads to any lan or wan machine it's with max. 128 kbits. When downloading, the speed is normal (100 mbps)

I have dummynet on all servers but I have removed all limits. There's no pipes for this servers.

The only thing different from old server is it's hardware. Maybe I was doing something wrong in the past too, but it was working.

I'm asking again, is there any change or difference in setting up ipfw+natd on freebsd amd64?

Thanks
 
Hello,

Nobody posted to this thread but no problem :) I want to update it so that someone, who has the same issue like me, can find some information here.

First, I'm sure that the problem is with FreeBSD 8.0 amd64. Because I have tested the same configurations on several i386 machine. Maybe it's a kernel or ipfw bug. I don't know. But I don't think that it's a configuration issue.

How I deal with that?

I have switched to PF. I've removed ipfw and set up pf instead of ipfw. I've configured nat and port redirection under pf.

My server is working very fast now. The problem had gone.

If you have the same problem with ipfw, I suggest you to use pf instead of ipfw to solve your problem easily. Yes, it's not solving ipfw, but it's solving our situation.

If that was a bug with ipfw; if developers fix it in the future, I can switch to ipfw again. Because I love it :)

To add, sorry for opening thread under network category. Now I think that it's related with firewalls. Moderators may move the topic to firewalls category. You can even remove this thread because I have self talked :)

May the source be with you!
 
I can concur, however I'm running i386 32bit (on a AMD X2 4800).

Behavior: as packets begin streaming, says a http webpage is called (responded to by apache22), the natd and dhcpd daemons start climbing rapidly in CPU. They max out at ~65% and ~45% respectively until they crush ssh and ultimately serial console folds, resulting in a manual hardware reboot to restore the server.

At first, using tcpdump -Xnvvvvi nfe0, I found bad tcp checksums on most packets. The eth cable and switch were found to be clear, so I thought the nic was responsible for the server quickly turning to thick sludge. After further troubleshooting, I wasn't so sure. Found another forum discussion indicating that turning off checksum offloads could fix the tcp bad checksum issue. Sure enough it did, but the same massive slowdown behavior still remained.

Thought maybe my northbridge was fragging up the bus. The server slowly would heal itself once I stopped the webpage from loading. CPU returned to nominal. Packet monitoring showed the problem only occurred when packets started passing rapidly. Something was bungling packet communications.

I found this thread. As soon as I shut down ipfw, the server was quick-snappy. No more issues. ipfw back on, issues come back. ipfw off, fixed. Ok, confirmed.

System info, if anyone is willing to dig into a potential bug fix. btw this is my first freebsd forum post. Been using bsd since bsdi, although I wouldn't say I'm an expert. If anyone needs more info, cli output, etc, please contact or post.

Code:
[root@phoenixxi ~]# uname -a
FreeBSD phoenixxi.localhost 8.1-RELEASE FreeBSD 8.1-RELEASE #2: Mon Oct  4 04:18:23 CDT 2010     
[email]root@phoenixxi.localh[/email]ost:/usr/obj/usr/src/sys/KERNEL_XI  i386

Code:
[root@phoenixxi ~]# cat /var/log/dmesg.today 
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE #2: Mon Oct  4 04:18:23 CDT 2010
    [email]root@phoenixxi.localhost[/email]:/usr/obj/usr/src/sys/KERNEL_XI i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2412.38-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f32  Family = f  Model = 23  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
  AMD Features2=0x3<LAHF,CMP>
real memory  = 2147483648 (2048 MB)
avail memory = 2095587328 (1998 MB)
ACPI APIC Table: <Nvidia AWRDACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7fef0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xd2002000-0xd2002fff irq 21 at device 2.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <NVIDIA nForce4 USB 2.0 controller> mem 0xfeb00000-0xfeb000ff irq 22 at device 2.1 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <NVIDIA nForce4 USB 2.0 controller> on ehci0
atapci0: <nVidia nForce CK804 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 6.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce CK804 SATA300 controller> port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xd400-0xd40f mem 0xd2001000-0xd2001fff 
irq 23 at device 8.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib1
nfe0: <NVIDIA nForce4 CK804 MCP9 Networking Adapter> port 0xc000-0xc007 mem 0xd2000000-0xd2000fff irq 21 at device 10.0 on pci0
miibus0: <MII bus> on nfe0
e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 1 on miibus0
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
nfe0: Ethernet address: 00:13:d4:82:ff:d9
nfe0: [FILTER]
pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci4: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 12.0 on pci0
pci3: <ACPI PCI bus> on pcib3
mskc0: <Marvell Yukon 88E8053 Gigabit Ethernet> port 0xb000-0xb0ff mem 0xd1000000-0xd1003fff irq 16 at device 0.0 on pci3
msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0
msk0: Ethernet address: 00:50:43:00:6d:7f
miibus1: <MII bus> on msk0
e1000phy1: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus1
e1000phy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
mskc0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci2: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci1: <ACPI PCI bus> on pcib5
acpi_tz0: <Thermal Zone> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x73 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (115200,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc8000-0xcbfff pnpid ORM0000 on isa0
ppc0: parallel port not found.
powernow0: <Cool`n'Quiet K8> on cpu0
powernow1: <Cool`n'Quiet K8> on cpu1
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based forwarding disabled, default to accept, logging disabled
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ad4: 15272MB <TS16GSSD25S S V090216> at ata2-master UDMA100 SATA 3Gb/s
SMP: AP CPU #1 Launched!
ugen0.1: <nVidia> at usbus0ugen1.1: <nVidia> at usbus1
uhub0: 
<nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Root mount waiting for: usbus1 usbus0
uhub0: 10 ports with 10 removable, self powered
Root mount waiting for: usbus1
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 10 ports with 10 removable, self powered
Trying to mount root from ufs:/dev/ad4s1a
nfe0: link state changed to UP
 
I have the solution with natd

Hi guys,

I had the same problem with natd and decided as follows:

Code:
ipfw add 1000 divert 8668 all from 192.168.0.0/24 to any via bce0

In my case, my localnet has the address 192.168.0.0/24.

My natd.conf:

Code:
dynamic yes
use_sockets yes
same_ports yes
unregistered_only yes
log yes
alias_address 222.111.222.222

With this, the upload returned to normal.
 
I was trying with IPFW + NATD but negative, no luck, finally I found this issue exposed by @jailed, that was my solution, thanks @jailed and all others that like FreeBSD.
Code:
my system
root@server:~ # uname -a
FreeBSD mydomain 10.0-RELEASE-p3 FreeBSD 10.0-RELEASE-p3 #0: Tue May 13 18:31:10 UTC 2014     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
in /etc/rc.conf
Code:
gateway_enable="YES"
ipv6_gateway_enable="YES"         #for ipv6

  pf_enable="YES"
  pf_flags=""
  pf_rules="/etc/pf.conf"
  pflog_enable="YES"
  pflog_logfile="/var/log/pflog"  # where pflogd should store the logfile
  pflog_flags=""                  # additional flags for pflogd startup

  obspamd_enable="YES"
  obspamlogd_enable="YES"

  ftpproxy_enable="YES"
In /etc/sysctl.conf
Code:
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
My guide
http://www.freebsd.org/doc/handbook/firewalls-pf.html
/usr/share/examples/pf/pf.conf
Thanks again to all that wrote their solutions here, in order to make it easy to solve problems.
 
Last edited by a moderator:
Back
Top