benchmarking network: faster one way?

Hi All,
When benchmarking with ttcp/iperf I find that I get very different results when sending and receiving between two directly connected devices, depending on who the sender is.

When sending from one device I get:

Code:
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> 192.168.1.2
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 0.19 real seconds = [B]84668.77 KB/sec[/B] +++
ttcp-t: 2048 I/O calls, msec/call = 0.10, calls/sec = 10583.60
ttcp-t: 0.0user 0.0sys 0:00real 47% 0i+0d 0maxrss 0+0pf 1340+0csw

But when sending from the other I get:

Code:
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> 192.168.1.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 0.32 real seconds = [B]51132.25 KB/sec[/B] +++
ttcp-t: 2048 I/O calls, msec/call = 0.16, calls/sec = 6391.53
ttcp-t: 0.0user 0.1sys 0:00real 33% 27i+1936d 1208maxrss 0+2pf 3173+7csw

Both devices are using Intel gigabit NICs. Hardware-wise neither of these systems should be exhausted by performing this benchmark. Both are sending from RAM, not disk, and neither CPU is overloaded. During the benchmark the network path does not change is not otherwise saturated.

In this case one device is running FreeBSD 7.2-RELEASE, the other is OS X 10.5. In the above example the slower transfer came from the FreeBSD box.

What would cause a 40% difference in throughput?
 
Have you checked the various counters with the following commands?

# netstat -ibd
# netstat -s

cheers,
honk
 
I will take a look at the suggestions later today and post back the results. Thanks.
 
While the data is interesting, I'm not sure what's relevant to my speed issue. If you see something I don't, kindly make a suggestion. The information below is from the FreeBSD box.

The output for netstat -s.

Code:
tcp:
	25577246 packets sent
		16566451 data packets (3345659095 bytes)
		147243 data packets (61678538 bytes) retransmitted
		7920 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		7601949 ack-only packets (0 delayed)
		0 URG only packets
		29035 window probe packets
		1059406 window update packets
		173162 control packets
	12741776 packets received
		5737260 acks (for 3343950919 bytes)
		1333516 duplicate acks
		190 acks for unsent data
		5759457 packets (1727528509 bytes) received in-sequence
		1164332 completely duplicate packets (24821089 bytes)
		1011 old duplicate packets
		4350 packets with some dup. data (875295 bytes duped)
		191293 out-of-order packets (188113464 bytes)
		4 packets (0 bytes) of data after window
		0 window probes
		211964 window update packets
		1458 packets received after close
		543 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		35 discarded due to memory problems
	134524 connection requests
	1704 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	886 ignored RSTs in the windows
	41215 connections established (including accepts)
	137328 connections closed (including 4767 drops)
		19272 connections updated cached RTT on close
		20399 connections updated cached RTT variance on close
		10359 connections updated cached ssthresh on close
	24460 embryonic connections dropped
	5350497 segments updated rtt (of 3174531 attempts)
	574720 retransmit timeouts
		1245 connections dropped by rexmit timeout
	29051 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	483 keepalive timeouts
		441 keepalive probes sent
		42 connections dropped by keepalive
	3107092 correct ACK header predictions
	5141417 correct data packet header predictions
	1704 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		1704 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	1704 cookies sent
	0 cookies received
	9003 SACK recovery episodes
	12219 segment rexmits in SACK recovery episodes
	5661419 byte rexmits in SACK recovery episodes
	100306 SACK options (SACK blocks) received
	207885 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
udp:
	248730 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	19 with no checksum
	0 dropped due to no socket
	38690 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	210040 delivered
	292208 datagrams output
	0 times multicast source filter matched
ip:
	13004249 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	1443 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	709 packets reassembled ok
	12990619 packets for this host
	12860 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	36 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	25885575 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header

And the output for netstat -ibd
Code:
Name    Mtu Network       Address              Ipkts Ierrs     Ibytes    Opkts Oerrs     Obytes  Coll Drop
em0    1500 <Link#1>      00:1b:21:56:49:fc 13035734     0 2782078507 25910562     0  767824332     0    0 
em0    1500 192.168.1.0   freenas           12959755     - 2583479984 25841183     -  403145278     -    -


Finally the output for ifconfig em0
Code:
em0: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST> metric 0 mtu 1500
	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether 00:1b:21:56:49:fc
	inet 192.168.1.111 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (1000baseTX <full-duplex>)
	status: active
 
Code:
1333516 duplicate acks

My money is on a duplex mismatch somewhere between the 2 boxes. Check the interfaces on the them and the switch ports they plug into and if possible try fixing everything at 1000MB/full and try again.
 
Thanks for the suggestion. Both of these devices are on the same router, not a switch, but I will double-check that full duplex is enabled on both ports.
 
Code:
1164332 completely duplicate packets (24821089 bytes)
191293 out-of-order packets (188113464 bytes)

Neither of those look particularly fantastic either.
 
I've hard-coded the duplex. The number of dups has decreased, but no change in speed.

Code:
139464 acks (for 735691221 bytes)
		3196 duplicate acks
 
Back
Top