I'm trying to figure out why FreeBSD cannot manage to achieve decent transfer rates in both directions as compared to Ubuntu. I'm performing my tests as follows:
Source: FreeBSD 9.2 AMD
Target: Ubuntu 14.04 x64
When testing speed with iperf, Source -> Target I'm getting about 20Mbps:
The same test in reverse (Target -> Source) the speed is fantastic:
Target -> Source:
The Target is capable of gigabit speed downloads from other hosts.
Traceroutes:
Source -> Target:
Target -> Source:
We thought there could just be a bad hop or perhaps it was hardware at fault, but when we run our iperf from the SAME hardware but booted to a stock Ubuntu 12.04 install, we get great/better/acceptable results:
(Ubuntu 12.04) Source -> Target:
So, it would seem that FreeBSD is very well tuned for Target -> Source traffic (nearly 1Gbps), but the Source -> Target direction is terrible (20Mbps). Since the RTT is the same and the # of hops is virtually unchanged, we don't see any reason the speed should be different outbound, and further we see that when we use Ubuntu -> Target, we get decent speed (100Mbps). Thus we conclude the hardware and hops must not the issue. So, what we're missing in our FreeBSD tuning that is making the outbound transfers so slow (while the inbound transfers are great)? And why is Ubuntu handling this speed so much better on the identical hardware?
Thanks
Source settings (FreeBSD):
Target settings (Ubuntu):
Source: FreeBSD 9.2 AMD
Target: Ubuntu 14.04 x64
When testing speed with iperf, Source -> Target I'm getting about 20Mbps:
Code:
iperf -t10 -P1 -i1 -c xxxxxxxx
------------------------------------------------------------
Client connecting to xxxxxxxxx, TCP port 5001
TCP window size: 2.16 MByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 3.50 MBytes 29.4 Mbits/sec
[ 3] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec
[ 3] 2.0- 3.0 sec 6.38 MBytes 53.5 Mbits/sec
[ 3] 3.0- 4.0 sec 2.12 MBytes 17.8 Mbits/sec
[ 3] 4.0- 5.0 sec 3.25 MBytes 27.3 Mbits/sec
[ 3] 5.0- 6.0 sec 4.25 MBytes 35.7 Mbits/sec
[ 3] 6.0- 7.0 sec 1.88 MBytes 15.7 Mbits/sec
[ 3] 7.0- 8.0 sec 4.12 MBytes 34.6 Mbits/sec
[ 3] 8.0- 9.0 sec 1.25 MBytes 10.5 Mbits/sec
[ 3] 9.0-10.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 3] 0.0-10.1 sec 31.8 MBytes 26.4 Mbits/secThe same test in reverse (Target -> Source) the speed is fantastic:
Target -> Source:
Code:
------------------------------------------------------------
Client connecting to xxxxxxxx, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 90.8 MBytes 761 Mbits/sec
[ 3] 1.0- 2.0 sec 104 MBytes 871 Mbits/sec
[ 3] 2.0- 3.0 sec 107 MBytes 900 Mbits/sec
[ 3] 3.0- 4.0 sec 96.0 MBytes 805 Mbits/sec
[ 3] 4.0- 5.0 sec 97.8 MBytes 820 Mbits/sec
[ 3] 5.0- 6.0 sec 102 MBytes 857 Mbits/sec
[ 3] 6.0- 7.0 sec 104 MBytes 873 Mbits/sec
[ 3] 7.0- 8.0 sec 104 MBytes 868 Mbits/sec
[ 3] 8.0- 9.0 sec 104 MBytes 873 Mbits/sec
[ 3] 9.0-10.0 sec 104 MBytes 871 Mbits/sec
[ 3] 0.0-10.0 sec 1014 MBytes 850 Mbits/secThe Target is capable of gigabit speed downloads from other hosts.
Traceroutes:
Source -> Target:
Code:
traceroute to xxxxxxx 64 hops max, 52 byte packets
1 6.978 ms 1.989 ms 2.002 ms
2 10.954 ms 0.983 ms
1.957 ms
3 52.189 ms 5.998 ms 22.044 ms
4 26.091 ms 22.056 ms 24.017 ms
5 21.492 ms 21.029 ms
6 21.047 ms 21.093 ms 21.998 ms
7 20.897 ms 20.744 ms 23.042 ms
8 20.699 ms 20.655 ms 20.526 msTarget -> Source:
Code:
traceroute to xxxxxx, 30 hops max, 60 byte packets
1 0.782 ms 0.761 ms 0.784 ms
2 1.072 ms 1.028 ms 1.002 ms
3 0.689 ms 0.665 ms 0.796 ms
4 42.513 ms 42.596 ms 42.568 ms
5 0.917 ms 0.895 ms 0.866 ms
6 20.209 ms 28.508 ms 28.507 ms
7 20.346 ms 20.352 ms 20.392 ms
8 30.392 ms 30.404 ms 30.387 ms
9 20.542 ms 20.720 ms 20.899 msWe thought there could just be a bad hop or perhaps it was hardware at fault, but when we run our iperf from the SAME hardware but booted to a stock Ubuntu 12.04 install, we get great/better/acceptable results:
(Ubuntu 12.04) Source -> Target:
Code:
$ iperf -t10 -P1 -i1 -w1M -c xxxxxxx
------------------------------------------------------------
Client connecting to xxxxxxxx, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 9.25 MBytes 77.6 Mbits/sec
[ 3] 1.0- 2.0 sec 10.2 MBytes 86.0 Mbits/sec
[ 3] 2.0- 3.0 sec 10.6 MBytes 89.1 Mbits/sec
[ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 5.0- 6.0 sec 10.8 MBytes 90.2 Mbits/sec
[ 3] 6.0- 7.0 sec 11.5 MBytes 96.5 Mbits/sec
[ 3] 7.0- 8.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 8.0- 9.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 0.0-10.0 sec 108 MBytes 90.7 Mbits/secSo, it would seem that FreeBSD is very well tuned for Target -> Source traffic (nearly 1Gbps), but the Source -> Target direction is terrible (20Mbps). Since the RTT is the same and the # of hops is virtually unchanged, we don't see any reason the speed should be different outbound, and further we see that when we use Ubuntu -> Target, we get decent speed (100Mbps). Thus we conclude the hardware and hops must not the issue. So, what we're missing in our FreeBSD tuning that is making the outbound transfers so slow (while the inbound transfers are great)? And why is Ubuntu handling this speed so much better on the identical hardware?
Thanks
Source settings (FreeBSD):
Code:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether f0:1f:af:11:e9:de
inet6 fe80::f21f:afff:fe11:e9de%bge0 prefixlen 64 scopeid 0x3
inet xxxxxxxx netmask 0xff000000 broadcast 82.255.255.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
kern.ipc.maxsockbuf: 16777216
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 128
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.nmbjumbo16: 3200
kern.ipc.nmbjumbo9: 6400
kern.ipc.nmbjumbop: 12800
kern.ipc.nmbclusters: 25600
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 21315584
kern.ipc.maxpipekva: 4080218112
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 152
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 60
kern.ipc.semmnu: 30
kern.ipc.semmns: 60
kern.ipc.semmni: 10
kern.ipc.semmap: 30
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 8192
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 33554432
kern.ipc.maxsockets: 25600
kern.ipc.numopensockets: 1281
kern.ipc.nsfbufsused: 0
kern.ipc.nsfbufspeak: 0
kern.ipc.nsfbufs: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.mssdflt: 1460
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 2263000
net.inet.tcp.recvspace: 2263000
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.v6mssdflt: 1024
net.inet.tcp.cc.available: newreno
net.inet.tcp.cc.algorithm: newreno
net.inet.tcp.hostcache.purge: 0
net.inet.tcp.hostcache.prune: 300
net.inet.tcp.hostcache.expire: 3600
net.inet.tcp.hostcache.count: 193
net.inet.tcp.hostcache.bucketlimit: 30
net.inet.tcp.hostcache.hashsize: 512
net.inet.tcp.hostcache.cachelimit: 15360
net.inet.tcp.read_locking: 1
net.inet.tcp.recvbuf_max: 16777216
net.inet.tcp.recvbuf_inc: 524288
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.insecure_rst: 0
net.inet.tcp.ecn.maxretries: 1
net.inet.tcp.ecn.enable: 0
net.inet.tcp.abc_l_var: 2
net.inet.tcp.rfc3465: 1
net.inet.tcp.rfc3390: 1
net.inet.tcp.rfc3042: 1
net.inet.tcp.drop_synfin: 1
net.inet.tcp.delayed_ack: 1
net.inet.tcp.blackhole: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.sendbuf_max: 16777216
net.inet.tcp.sendbuf_inc: 16384
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.tso: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.slowstart_flightsize: 1550
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.reass.overflows: 481332
net.inet.tcp.reass.cursegments: 4
net.inet.tcp.reass.maxsegments: 1680
net.inet.tcp.sack.globalholes: 0
net.inet.tcp.sack.globalmaxholes: 65536
net.inet.tcp.sack.maxholes: 128
net.inet.tcp.sack.enable: 1
net.inet.tcp.inflight.stab: 20
net.inet.tcp.inflight.max: 1073725440
net.inet.tcp.inflight.min: 6144
net.inet.tcp.inflight.rttthresh: 10
net.inet.tcp.inflight.debug: 0
net.inet.tcp.inflight.enable: 0
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.pcbcount: 461
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.log_debug: 0
net.inet.tcp.minmss: 216
net.inet.tcp.syncache.rst_on_sock_fail: 1
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.count: 4294967277
net.inet.tcp.syncache.cachelimit: 15360
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncookies_only: 0
net.inet.tcp.syncookies: 1
net.inet.tcp.timer_race: 1
net.inet.tcp.rexmit_drop_options: 1
net.inet.tcp.finwait2_timeout: 60000
net.inet.tcp.fast_finwait2_recycle: 0
net.inet.tcp.always_keepalive: 1
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.rexmit_min: 30
net.inet.tcp.msl: 30000
net.inet.tcp.nolocaltimewait: 0
net.inet.tcp.maxtcptw: 5120Target settings (Ubuntu):
Code:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:840363516 errors:0 dropped:0 overruns:0 frame:0
TX packets:152325250 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2086148084848 (2.0 TB) TX bytes:64233189137 (64.2 GB)
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 32768
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 256
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_mem = 524288 524288 524288
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = -1
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_workaround_signed_windows = 0
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = pfifo_fast
net.core.dev_weight = 64
net.core.flow_limit_cpu_bitmap = 00
net.core.flow_limit_table_len = 4096
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 1000
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 524288
net.core.rmem_max = 33554432
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 128
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 524288
net.core.wmem_max = 33554432
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
