Setting up router -- stalled

[jurrie]

Hi,

today I got my fiber connection replacing my dsl line. I have always used a nice Linksys DSL modem / router for my dsl line, but it won't work with the fiber connection (it lacks an ethernet uplink port). I decided to use my BSD home server to be my router for the time being.

I have used this guide (http://freebsd.rogness.net/redirect.cgi?basic/gateway.html) to set up my server as a gateway and configured it to use NAT as well. Things work fantastic and I can do all my stuff using my desktop perfectly.

The problem, however, lies with the server. When I try to scp from the server, or serve a page using apache, the connection stalls. Uploading using my desktop works fine, but on the server it uploads a few KB (increments of 64) and then stalls for a while.

Being somewhat of a network nitwit (I can set up / maintain basic things, but when things get advanced, I drop out), I was hoping someone here could direct me to a fix.

To illustrate:

(Fiber modem)----<nfe0>(BSD Server)<em0>----(Switch)----(Desktop)

nfe0 is set to use DHCP (to work with my fiber connection) and em0 is set to 192.168.1.2 (legacy), with subnet mask 255.255.255.0, using my ISP's DNS.

My desktop is set to 192.168.1.3, subnet 255.255.255.0, also using my ISP's DNS.

The console prints no errors whatsoever. Nothing else I could say to provide more info comes to mind, so if I left something out which is required to answer my question, please let me know.

Thanks very much in advance.

[honk]

When I try to scp from the server, or serve a page using apache, the connection stalls. Uploading using my desktop works fine, but on the server it uploads a few KB (increments of 64) and then stalls for a while.

You mean serving for internet clients? It works for internal clients? Just an idea, I've an apache reachable from the internet and my provider drops connections after transfer of about 5MB in upstream direction if my servers port is 80 or 443. Moving my server services to a high-port helps, the conections from the clients don't break after 5MB.

I propose to use tcpdump to debug this.

[jurrie]

It's not just serving to clients, it's also using scp myself. I had a a friend connect to http://<my ip>/ where I had a phpinfo() in an index.php. He did see the output, though it took ages for the complete page to transfer. When I put a file there, he got it, verrrrry sloooowly. When scp'ing from the server to my web host, a bit was uploaded and then the -stalled- message appeared.

It's not an ISP issue, since I can use scp from my desktop to copy to my web host and it flies at around 650k/s. Speedtest.net also gives me around 18Mbit in upstream (all this using the server as gateway/router).

I think the server is somehow confused as to which network card it should use to upload to the outside network. But that's just my wild nitwit guess.

[honk]

Not sure if I understand you exactly...
When your internal desktop transfers data >>through<< your FBSD gateway/webserver to your (any) internet webhost, it's ok? And it's also ok if you transfer from your internal desktop to your FBSD gateway/webserver (within the local net)?? But from outside(internet) to your FBSD gateway/webserver its slow? If so it might be an MTU issue. Please post the output of

ifconfig -a
netstat -rn
netstat -i

And of course your firewall and NAT configuration. You might also check how your firewall handles icmp.

Try to figure out the MTU on your fiber connection by pinging the first hop like this:

# ping -s 1472 -D 1.2.3.4
PING 1.2.3.4 (1.2.3.4): 1472 data bytes
1480 bytes from 1.2.3.4: icmp_seq=0 ttl=62 time=10.547 ms
1480 bytes from 1.2.3.4: icmp_seq=1 ttl=62 time=10.464 ms
1480 bytes from 1.2.3.4: icmp_seq=2 ttl=62 time=10.158 ms
...
# ping -s 1473 -D 1.2.3.4
PING 1.2.3.4 (1.2.3.4): 1473 data bytes
ping: sendto: Message too long


Try different values for the packet size -D until you find the largest value before your don't get an answer back or a "Message too long".

[jurrie]

Thanks for replying again. I'll try to explain it once more before I will post the information you requested.

* Desktop -> uses server as gateway -> no problems whatsoever
* Server -> login via terminal -> scp to the internet -> stalled
* Remote user -> connects to my home IP via http -> receives data very slow

About the pings. I tried several times. "ping -s 1472 -D" works like a charm. Any number above the 1472 results in 100% packet loss. On rare occasions I do receive a ping back, but the lowest packet loss I've seen is 86.4%.

Output of the commands:

# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0e:0c:b6:0f:85
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM, TSO4>
ether 00:1d:7d:d7:58:16
inet 94.229.49.55 netmask 0xfffffe00 broadcast 94.229.49.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000

# netstat -rn
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 94.229.48.1 UGS 0 3278 nfe0
94.229.48.0/23 link#2 UC 0 0 nfe0
94.229.48.1 00:0e:00:00:00:01 UHLW 2 0 nfe0 469
94.229.48.23 00:0e:00:00:00:01 UHLW 1 4 nfe0 1073
94.229.49.55 00:1d:7d:d7:58:16 UHLW 1 40 lo0
94.229.49.255 ff:ff:ff:ff:ff:ff UHLWb 1 86 nfe0
127.0.0.1 127.0.0.1 UH 0 40 lo0
192.168.1.0/24 link#1 UC 0 0 em0
192.168.1.2 00:0e:0c:b6:0f:85 UHLW 1 16 lo0
192.168.1.3 00:18:f3:74:8c:a3 UHLW 1 60235 em0 825
192.168.1.255 ff:ff:ff:ff:ff:ff UHLWb 1 45 em0

Internet6:
Destination Gateway Flags Netif Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0


# netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
em0 1500 <Link#1> 00:0e:0c:b6:0f:85 85568 0 60840 0 0
em0 1500 192.168.1.0 merry 84819 - 59963 - -
nfe0 1500 <Link#2> 00:1d:7d:d7:58:16 1585 0 1803 0 0
nfe0 1500 94.229.48.0/2 5ee53137.ftth.con 680 - 1838 - -
plip0 1500 <Link#3> 0 0 0 0 0
lo0 16384 <Link#4> 96 0 96 0 0
lo0 16384 fe80:4::1 fe80:4::1 0 - 0 - -
lo0 16384 localhost ::1 0 - 0 - -
lo0 16384 your-net localhost 40 - 40 - -


Perhaps these outputs point to an obvious mistake in my settings, but they mean nothing to me.

My firewall settings (I hope I did it properly):

# ipfw -a l
ipfw: DEPRECATED: 'l' matched 'list' as a sub-string
00050 3300 2109884 divert 8668 ip4 from any to any via nfe0
00100 192 15274 allow ip from any to any via lo0
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
65000 151731 140144932 allow ip from any to any
65535 4 796 deny ip from any to any


And natd settings (only a port forward for my torrent client on my desktop):

# cat /etc/natd.conf
port 8668
interface nfe0
redirect_port tcp 192.168.1.3:6781 6781
redirect_port udp 192.168.1.3:6781 6781

I hope this clears things up. Thanks in advance for your help and effort.

[jurrie]

Okay, the problem is solved now. Thanks to honk for helping me via email!

The evildoer was the nvidia nic (nfe0). Several topics on the web pointed to it performing badly on several occasions. It was suggested to me that I should turn off tcp checksum offloading and tcp segmentation offloading on the nvidia nic, using "ifconfig nfe0 -txcsum -rxcsum -tso4". This, however, did not work for me (though it apparantly worked for others).

Then we tried the simplest of all solutions: switch the internal and external nics. Now I use my onboard nvidia nic for the internal network and my trusty intel pci nic for the external network. Things work like a charm now! Full blazing upload speed and no noticable problems on my internal network, now being served by the nvidia one.

I'm a happy camper now. Thanks again, honk!

[RaZwaN]

i have the same problem.
can you help me too?
i used the txcsum -rxcsum -tso4 and it works for now but i'm wondering if it is another solution for this problem.

thanks!

[jurrie]

It's been a while, but I recall the nvidia nic having the problems when it's not operating on 1Gbit. Before I switched the nics, the nvidia one was communicating with my fibre line (35Mbit) and my intel nic with my internal network (1Gbit). After switching the nvidia nic to the 1Gbit network, it stopped acting up. I'm afraid that's all I can say about that.