Broadcom NetXtreme II BCM5708

[nbari]

I have a collocated server the one is running with the latest stable FreeBSD 8.1 amd64 on a DELL PowerEdge 2900II with Broadcom Nextreme II BCM5708 network cards.

People that use to download files from the server start to notice that the download speed of the files was slower, I later make some tests and this was true, the server in theory is attached to a 100mbps port and from other servers with the same capability I can only download at max rates of 400K.

I contact the data center make make some tests and told me that this issue was from my side, from their internal network the file downloads at excellent speeds (12MB/s) the only problem exists only when downloading the file from external networks.

This issue extended for days, they told me that did exhaustive tests and that can confirm that was an issue from my side.

One of the test they made was to boot the server with a knoppix cd live they configured the network put a testfile for download, and the download speeds for the external network now where much better than when running from the OS on the server, from their internal lan the speeds where the same but from the internet a notable difference was appreciated.

later the server was rebooted again and loaded the installed OS (FredBSD 8.1) but again like if where magic the speeds rates now where only good from the internal network rather than the outside network.

I have already checked the server, make some ssh tunnels through another server hosted on the same datacenter and got good speeds, the only problem that i notice is that there is something limiting the outgoing bandwidth since internally the speeds are excellent i can not understand what else could be wrong.

Any suggestions for fixing this ?

any recommendations ?

thanks in advance.

 

[SirDice]

If you get good speeds on the same network then it's probably not something that's related to your server.

It could be a routing issue (routing to the wrong gateway), you may want to check that.

 

[Terry_Kennedy]

later the server was rebooted again and loaded the installed OS (FredBSD 8.1) but again like if where magic the speeds rates now where only good from the internal network rather than the outside network.

I have already checked the server, make some ssh tunnels through another server hosted on the same datacenter and got good speeds, the only problem that i notice is that there is something limiting the outgoing bandwidth since internally the speeds are excellent i can not understand what else could be wrong.

Was the other server also a FreeBSD system running the same FreeBSD version? Or something else?

It might be some TCP/IP network setting (like a small window size) that would be very sensitive to networking latency (so a client testing from the same LAN would see better performance than one elsewhere on the Internet). Look in /etc/sysctl.conf and see if there are any "net..." tunables set in there.

I haven't felt the need to change any of those tunables from the defaults, and I have servers with much faster network connections, so I don't think the defaults are broken. I just did a test with a Dell PowerEdge R300 with an add-in BCM5708 card located about 14 hops from my home net connection, and I got 42Mbit/sec (I have a 50Mbit/sec cable modem net connection, and there was other stuff going on at the same time).

If none of that helps, you may want to see if there's a Dell update for your 5708 card - on the R300, they've posted some firmware updates for that card. There haven't been any recent firmware changes that I'd say were performance-related, but you may have a very old card. Unfortunately, those Dell firmware update utilities don't run under FreeBSD (only Windows and Linux). However, you can boot from a Windows 7 installation DVD and select "Repair Windows installation" and then "Command prompt" and the Dell updaters will run from there (put them on a USB memory stick so you can have both the Windows disc and the Dell updates available at the same time). You don't need to actually install Windows, just boot from the DVD.

 

[noisex]

Try to some sysctl TCP tuning...sometimes on 1Gbps/100Mbps links TCP window size, send/rcv buffer size is to small.

For an example:

kern.ipc.somaxconn=2048
kern.ipc.nmbclusters=32768

kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576
net.inet.udp.recvspace=1048576
net.inet.tcp.rfc1323=1
net.inet.tcp.sack.enable=1
net.inet.tcp.path_mtu_discovery=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.hostcache.expire=1