Can't saturate 100Mbit network

G

glug

I'm experienced some strange behavior with my FreeBSD 9-RELEASE server related to network speeds. I have my server set up with a raidz2 serving media via NFS and SMB. My FreeBSD server is sitting behind a 1Gbit switch along with a Win7 computer. I also have a Linux box running XBMC that's connected to the switch via a set of MoCA network adapters. The MoCA adapters limit the network speed to the XBMC computer to 100Mbit/sec.

I haven't been able to stream 1080P video from the FreeBSD server to the XBMC box without a lot of stuttering. If I stream the same video from the Win7 machine, the playback is fine. I performed some testing, and here's what I found:

FreeBSD server to Win7 machine (1Gbit connection): 50 - 70MB/sec. The speed isn't really consistent, but it never dips below 50MB/s.

FreeBSD server to XBMC: ~4-6MB/sec. The connection isn't very steady and occasionally dips below 4MB/sec. Both NFS and SMB yield similar results, though the SMB is even slower than the NFS.

Win7 machine to XBMC: 10.5MB/sec. This connection is very steady and doesn't fluctuate much at all.

I don't understand why my FreeBSD server can't saturate a 100Mbit share to my XBMC computer like the Win7 machine. After all, it can serve between 50 and 70MB/sec on a gigabit connection. I've experimented with both NFS and SMB after following all the tuning guides for both. Any ideas?
 
Things to check:

1) Obtain the MTU settings on both sides, then check it by issuing the following command on the FreeBSD machine, and then a similar one on the Linux box.

$ ping -Dc1 -s1472 192.168.x.y

Replace the packet size 1472 with your actual MTU-28, and 192.168.x.y by the actual target address,


2) Check that both sides have the same TCP delayed acknowledgment behaviour. A mismatch can completely deteriorate the transfer rate.

On FreeBSD you would set the following sysctl to either 1 or 0. 1 is the default. [CMD="example #"]sysctl net.inet.tcp.delayed_ack=0[/CMD]. I do not know the respective setting/command on Linux though.

3) There are some threads on this forum regarding optimizing the smb for transfer speed. This involves increasing some buffer sizes, and using asynchronous IO, I do not remember the bloody details though, and perhaps others may help out here.
 
Are they recent 4k sector hard drives? If your drives/partitions are not 4k aligned, and the drives are 4k sectors, performance degrades very very badly.
 
First check your network card media settings:
Code: 
[cmd=#]ifconfig re0 | egrep '(flags|options|media)'[/cmd]
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        [color=blue]media[/color]: Ethernet autoselect (1000baseT <full-duplex>)
Here it confirms that the connection to the switch is 1000mbit full-duplex.

Then you can look for input and output errors as well as collisions with netstat(1):
Code: 
[cmd=$]netstat -in[/cmd]

Name    Mtu Network       Address              Ipkts [color=blue]Ierrs[/color]    Opkts [color=blue]Oerrs  Coll[/color]
bge0   1500 <Link#1>      00:1d:09:ff:2d:0a 26944124     0 33562872     0     0
bge0   1500 176.122.10.2  176.122.10.2      25376145     - 33484461     -     -
In a switched network you should see no collisions.

Another useful option for netstat(1) is -s to view interface statistics for a protocol family:
Code: 
[cmd=$]netstat -s -p tcp[/cmd]

tcp:
        32124812 packets sent
                25906079 data packets (32891796988 bytes)
                200548 data packets (256655583 bytes) retransmitted
                8458 data packets unnecessarily retransmitted
                79 resends initiated by MTU discovery
                4029607 ack-only packets (281938 delayed)
                0 URG only packets
                328 window probe packets
                270079 window update packets
                2038566 control packets
        24602953 packets received
                17323094 acks (for 32630354928 bytes)
                1546709 duplicate acks
                21683 acks for unsent data
                5202212 packets (2374310148 bytes) received in-sequence
                87044 completely duplicate packets (8424353 bytes)
                1748 old duplicate packets
                157 packets with some dup. data (71413 bytes duped)
                6362 out-of-order packets (3298204 bytes)
                2747 packets (0 bytes) of data after window
                0 window probes
                511538 window update packets
                1594 packets received after close
                66180 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
[snip]
You have to to look at retransmissions, duplicates, bad checksums etc.

In this case, a not so busy web server, the percentage of retransmitted data packets is about 0.77 %
Code: 
[cmd=$]echo $SHELL
/bin/sh[/cmd]

[cmd=$]kal(){ echo "scale=4 ; $@" | bc ;}[/cmd]

[cmd=$]kal '200548/259060.79'[/cmd]
.7741

The percentage of duplicate acks is much higher, about 8.9%.
Code: 
[cmd=$]kal '1546709/173230.94'[/cmd]
8.9285

Check the man page to see how you can reset these netstat(1) counters to 0.
 
Thanks to everyone for your helpful suggestions. I checked into the sysctl parameters as suggested by @rolfheinrich. Using iperf as a benchmark tool I managed to get the FreeBSD to XBMC connection up to 9.5MB/sec. That still wasn't as good as the 10.5MB/sec that I was getting from my Win7 to XBMC, but it should be fast enough to stream my 20Mbit/sec video. Oddly though, playback still stuttered.

Using iperf again, I found that in UDP mode, I could get about 11.4MB/sec out of the connection, but I had packet loss of 4%. I suspect the high packet loss is due to the MoCA network to the XBMC machine. I ran iperf on the Win7 to XBMC connection and saw the same 4% packet loss. I began to suspect that the handling of this packet loss may have something to do with my problems.

Finally, I wanted to rule out the possibility that the network card itself wasn't at fault. To do this, I created a virtual machine on another server with the same FreeBSD 9-RELEASE, RAM, CPUSs, and configuration as my physical FreeBSD server. To my surprise, it worked great! The video streamed to my XBMC box from the FreeBSD virtual machine didn't stutter at all. The only real difference between the two was the network adapter. My FreeBSD server has this network card:
Code: 
nfe0: <NVIDIA nForce MCP79 Networking Adapter>

The FreeBSD virtual machine has this network card:
Code: 
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3>

So I now suspect that there is a problem with either the network adapter itself or the FreeBSD driver for it. If there are no other suggestions, I'll try an Intel PCI network adaptor and see if that fixes the problem.
 
Last edited by a moderator:
Update: I found an old D-Link 10/100 network card in the parts bin. I through it in the server, and now everything is working great.
 
You must log in or register to reply here.
Back
Top