Download limitations on FreeBSD Router?

pothovent · #1 July 17th, 2012, 15:42

Hello to everyone!

Even though I am a rather BSD "greenhorn", I've used FreeBSD as a router for the better part of a decade. The love I have for the clarity of PF is something that must have been felt by some of you out there as well. However, dark clouds are forming over this paradise that is called my home. (Sorry for being dramatic, I've been searching around for too long now.)

Just to paint a picture: I'm probably a rather "default" FreeBSD server / router (AMD 64, FreeBSD 8.2) user. I have connected my router directly to the cable modem on one interface (nfe0) and another interface (em0) is connected to my internal network. On this server I run samba, a transmission daemon and some more services which are probably not relevant for the discussion.

The problem: Recently I upgraded my bandwidth and discovered that my router was somehow limiting my downstream on the clients. While performing a speed test provided by my ISP (and checked with http://www.speedtest.net/) I discovered that the upstream on my clients are relatively healthy. (Being that I officially get 6 Mbps and the tests show anywhere from 4.X to 5.Y). The 60 Mbps downstream however is completely stuck at 8-9 Mbps. Connecting a device (e.g. my laptop) directly to the cable model shows a dramatic increase to around 54 Mbps. Conclusion: It must be somewhere in my router.

"Googling" around I discovered that this could be possible due to the limitations in the TCP protocol. Another discussion states that PF "limits" it somehow, but as far as I know you can't limit your incoming bandwidth. However, although my background is in computer science, I completely lack all the skills and know how on how to proceed further in discovering what's going on.

Therefor I would like to call out to the BSD community. The forums have been a great guideline for many years for me, but this particular problem is something I haven't seen yet. (Or I completely missed, in that case: Show me the link and I'll be forever grateful.)

If this problem is familiar to someone, please let me know. If someone would like to instruct a greenhorn on how to investigate this problem, please let me know what you would like to see and I'll try to provide it as good as possible.

Kind regards,

Tristan Pothoven

PS. Perhaps completely unrelated, perhaps not: Running a download in an application (e.g. Steam) completely ruins the latency from and to my server. A 1 MB/sec download provides a 400-500 ms delay for a reason I have yet to discover.

**DutchDaemon** · #2 July 17th, 2012, 16:01

We'd probably have to see your ruleset. I've pushed 100Mbit+ (full-duplex) through PF on a consumer desktop PC with fiber interfaces.

pothovent · #3 July 17th, 2012, 16:12

Thanks for the quick response.

I was surprised as well, I can imagine something with "network balancing", but I'd like to be able to control it instead of just "accepting it's there".

Here's my PF ruleset (assuming you were talking about that ruleset):

Code:

# Macros
ext_if="nfe0"
int_if="em0"
tristan="192.168.0.10" # My own system for port forwarding

# Options
set skip on lo0

# Normalization
scrub in all

# Translation
nat pass on $ext_if inet from $int_if:network to any -> ($ext_if)

# Redirection
#rdr pass on $ext_if inet proto tcp from any to ($ext_if) port 55555 -> $tristan #formerly used for torrent, now replaced with Transmission
#rdr pass on $ext_if inet proto udp from any to ($ext_if) port 55555 -> $tristan

# Filtering
#pass in quick all
#pass out quick all
block in log all
block out log all
pass in quick on $int_if all
pass out quick on $int_if all
pass in on $ext_if proto {icmp} all
pass out on $ext_if proto {icmp} all
pass out on $ext_if inet  proto tcp from ($ext_if) to any flags S/SA modulate state
pass out on $ext_if inet  proto udp from ($ext_if) to any keep state

block return in log on $ext_if proto {tcp,udp} from any to ($ext_if)
pass in on $ext_if inet  proto tcp from any to ($ext_if) port 22 flags S/SA modulate state

#allow all over 49152
pass in on $ext_if inet  proto tcp from any to ($ext_if) port >= 49152 flags S/SA modulate state

It's based on an online guide which can be found on http://www.bsdguides.org/guides/free..._router_pf.php

Thanks in advance.

**SirDice** · #4 July 17th, 2012, 16:25

Your pf.conf looks fine, nothing that would warrant the speed limit.

Looking at your speeds though, is it possible the connection between your FreeBSD machine and the router is limited to 10Mbit?

**DutchDaemon** · #5 July 17th, 2012, 16:26

If you don't do any filtering on $int_if, simply set a skip on it.
Put the busiest, most referenced rule at the top
Put quick on every rule you possibly can.
Conflate identical pass in quick and pass out quick rules to pass quick

Don't forget to run pfctl -sr to see what your actual expanded ruleset really looks like. You may find ways to economize more.

**wblock@** · #6 July 17th, 2012, 16:51

Speeds reported by speedtest.net are very dependent on the server. Try several.

pothovent · #7 July 17th, 2012, 16:53

Quote:

Originally Posted by SirDice

Your pf.conf looks fine, nothing that would warrant the speed limit.

Looking at your speeds though, is it possible the connection between your FreeBSD machine and the router is limited to 10Mbit?

Ehm. Perhaps I didnt clarify myself: my FreeBSD machine (aka: router) is directly connected to the cable modem. Connecting a laptop using the same cable improved the results quite a bit. Or am I completely missing your point?

**SirDice** · #8 July 17th, 2012, 16:56

Quote:

Originally Posted by pothovent

Ehm. Perhaps I didnt clarify myself: my FreeBSD machine (aka: router) is directly connected to the cable modem. Connecting a laptop using the same cable improved the results quite a bit. Or am I completely missing your point?

I understood that. The interface used to connect to the cable modem may be "fixed" at 10Mbit.

pothovent · #9 July 17th, 2012, 17:40

Quote:

Originally Posted by SirDice

I understood that. The interface used to connect to the cable modem may be "fixed" at 10Mbit.

Ah, you have a good point there. Looking at my ifconfig:

Code:

tristan@server:~$ sudo ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:1b:21:56:7a:d8
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8210b<RXCSUM,TXCSUM,VLAN_MTU,TSO4,WOL_MAGIC,LINKSTATE>
        ether 90:e6:ba:d9:ca:78
        inet 77.250.58.217 netmask 0xffffff00 broadcast 255.255.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

I suspect that that is not the problem.

And in response to the earlier suggestion of pfctl -sr (which indeed gives quite a lot of less clutter):

Code:

tristan@server:~$ sudo pfctl -sr
scrub in all fragment reassemble
block drop in log all
block drop out log all
pass in quick on em0 all flags S/SA keep state
pass out quick on em0 all flags S/SA keep state
pass in on nfe0 proto icmp all keep state
pass out on nfe0 proto icmp all keep state
pass out on nfe0 inet proto tcp from (nfe0) to any flags S/SA modulate state
pass out on nfe0 inet proto udp from (nfe0) to any keep state
block return in log on nfe0 proto tcp from any to (nfe0)
block return in log on nfe0 proto udp from any to (nfe0)
pass in on nfe0 inet proto tcp from any to (nfe0) port = ssh flags S/SA modulate state
pass in on nfe0 inet proto tcp from any to (nfe0) port >= 49152 flags S/SA modulate state

Now I must admit that my PF knowledge is really rusty and the suggestions about skipping and "quickening" need some more work.

For instance, do I see correctly that the following two lines:

Code:

pass in on nfe0 proto icmp all keep state
pass out on nfe0 proto icmp all keep state

can be replaced by:

Code:

pass quick nfe0 proto icmp all keep state

Or am I looking in a completely different direction here?

pothovent · #10 July 17th, 2012, 17:59

Incidentally, in response to DutchDaemon's comment. I've made some modifications to pf.conf:

Code:

<omitted first few lines>

# Options
set skip on lo0
set skip on $int_if

<omitted same items again>

# Filtering
#pass in quick all
#pass out quick all
block in log all
block out log all
pass in quick on $int_if all
pass out quick on $int_if all
pass in quick on $ext_if proto {icmp} all
pass out quick on $ext_if proto {icmp} all
pass out quick on $ext_if inet  proto tcp from ($ext_if) to any flags S/SA modulate state
pass out quick on $ext_if inet  proto udp from ($ext_if) to any keep state

block return in log on $ext_if proto {tcp,udp} from any to ($ext_if)
pass in quick on $ext_if inet  proto tcp from any to ($ext_if) port 22 flags S/SA modulate state

#allow all over 49152
pass in quick on $ext_if inet  proto tcp from any to ($ext_if) port >= 49152 flags S/SA modulate state

Which resulted in the next output:

Code:

tristan@server:~$ sudo pfctl -sr
scrub in all fragment reassemble
block drop in log all
block drop out log all
pass in quick on em0 all flags S/SA keep state
pass out quick on em0 all flags S/SA keep state
pass in quick on nfe0 proto icmp all keep state
pass out quick on nfe0 proto icmp all keep state
pass out quick on nfe0 inet proto tcp from (nfe0) to any flags S/SA modulate state
pass out quick on nfe0 inet proto udp from (nfe0) to any keep state
block return in log on nfe0 proto tcp from any to (nfe0)
block return in log on nfe0 proto udp from any to (nfe0)
pass in quick on nfe0 inet proto tcp from any to (nfe0) port = ssh flags S/SA modulate state
pass in quick on nfe0 inet proto tcp from any to (nfe0) port >= 49152 flags S/SA modulate state

According to the speed tests, still no improvement. I've tried reloading the pf.conf with "pfctl -f /etc/pf.conf" (which is the correct way as far as I know), but perhaps I've done something wrong.

Any advice on how to proceed would be more than welcome.

Kind regards,
Tristan

pothovent · #11 July 18th, 2012, 07:34

Hello everyone,

This morning I had a proper line of thought and decided to retest my network using different NIC's in the clients.

Suddenly everything worked, at high speeds.

Now, this is a little shameful to say, it appears that a certain not-by-name mentionable non-FreeBSD OS was limiting my NIC at 10 Mbs. (And yes, 3 days ago there was an "automatic driver update" available on that device. And yes, it happened on both clients I tried earlier.) Restoring the driver was the solution and now all is well in this paradise. (I do need to upgrade to cat-6 if I want to use gigabyte tho..)

In conclusion I must admid to SirDice: You were completely correct! But I was looking at the wrong NIC.

For now: Thanks to everyone for your quick and adequate response. And sorry to waste your time.

**SirDice** · #12 July 18th, 2012, 09:27

Quote:

Originally Posted by pothovent

(I do need to upgrade to cat-6 if I want to use gigabyte tho..)

Depending on the distance and the environment CAT-5e is usually enough. CAT-6 is still quite expensive.

Quote:

In conclusion I must admid to SirDice: You were completely correct! But I was looking at the wrong NIC.

I sometimes scare myself

pothovent · #13 July 18th, 2012, 09:31

It's about 3-5 meters.

I've tried forcing the NIC to gigabit, but it just wont work. Luckily I happen to have a cat 6 cable lying close, so a test is easily performed.

Anyhow, I think the mystery is solved here.

Thanks again.

gpeskens · #14 August 21st, 2012, 15:01

is your switch/access point supporting gigabit ?

really on those distances the regular cat5e (those with 8 wires, not those 4 wired ones that casema/ziggo used to send out) should really be enough.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ld limitations?	xiphos	Installation and Maintenance of FreeBSD Ports or Packages	0	April 6th, 2012 20:18
download freebsd	mohamed-taher	General	3	December 22nd, 2011 13:43
[IPF] Connecting Cisco ASA VPN through FreeBSD 8.2 router vs FreeBSD 7.4 router	gilcel	Firewalls	0	July 12th, 2011 10:40
Limitations of zip and unzip command on 32 bit machine	Rax	Installation and Maintenance of FreeBSD Ports or Packages	2	March 3rd, 2011 07:50
Signature limitations	jem	Feedback	2	October 18th, 2010 00:03