FreeBSD Network Troubleshooting
Learning more about network performance and the power tools available in FreeBSD for maximum network debugging
FreeBSD has a full set of debugging features, and the network stack is able to report a ton of information. So much that it can be hard to figure out what is relevant and what is not.
There are a number of things we can mean when we say a network is performing well and they don’t all have to align for a network to get a good score for a particular application. The nature of the workload is an important determinant of what is considered good. Network performance is dominated by two components that exist in symbiosis, delay and bandwidth.
Delay is the amount of time it takes for traffic to transverse the network. Most commonly we will talk about the round-trip time or ‘ping’, this measures the delay out and back to a remote host. Unless you are in a very special network (such as a VSAT link with dial up return) you can assume that the out and back delays are the same and that the one-way delay is half of the RTT. In reality, out and back routes very often differ, i.e. one path is faster than the other. Delay controls how responsive an application or protocol can be. The time between an action and the response to that action is referred to as the ‘latency’. We measure delay in milliseconds (ms).
Network capacity is the network’s ability to carry traffic over time; we normally use the term bandwidth when talking about the rates a network can send at, and in most cases these two terms are interchangeable. If you ever deal with radio systems it can be easy to get confused. Capacity governs the amount of information we can transfer in an amount of time (we use one second) and we typically talk about bandwidth in Megabit per second (Mbps) for links and Megabytes per second (MB/s) for application measures. The traffic a host or application can process in a second is also called its ‘throughput’.
We use the bandwidth and the delay together to estimate the size of a network with something called the Bandwidth Delay Product (BDP). The size of the BDP is important when calculating buffer sizes for transfers and is an area to tune. BDP governs a lot of the functionality of the congestion controller and hence the transfer speeds you can have between two hosts.
Protocols on the Internet are expected to share the available bandwidth fairly. Protocols that offer reliable delivery like TCP and QUIC use congestion controller to estimate how much bandwidth (the BDP) is available and react quickly by reducing sending rates when loss or reordering is detected. Classical congestion control algorithms grow and recover based on the number of packets that are successfully delivered at each RTT, meaning that large delays can have an impact on protocol performance.
Internet service is provided on a best effort basis, the network does not guarantee delivery of any particular packet. This is an optimization and simplification that has made our current Internet possible, but it means that errors are part of the default operating mode of the network. In a best effort network, we expect to see packet loss and protocols are either agnostic to it or have mechanisms to work around it. Protocols that offer reliable delivery, such as TCP, struggle when there is too much packet loss. High levels of packet loss (>1%) might be a sign of a network issue and will have a serious impact on transfer speeds.
Our Internet was built to be flexible to advances in host and link technology. In the 70s the routers of the network (at that point called imps) could convert between different byte orders and sizes (ever seen a non 8 bit byte?). We quickly realised that this put too much complexity into the network and made things adhere to a single standard. Link sizes were not standardised and instead we added mechanisms for hosts to learn the size of the largest packets that could be carried on a network path. When these mechanisms don’t work or aren’t usable, IPv4 routers are able to fragment packets. Unfortunately, fragments are expensive to process and require a lot of memory to reassemble and process on end hosts. For this reason, fragmentation is considered fragile and an increase in the number of fragments a host sees can indicate errors on the network.
Protocols: Hosts use many different protocols to communicate with each other on the internet and we layer these on top of each other to provide richer services to applications.
For performance we are going to concern ourselves with the network characteristics that affect the lower level protocols that carry communication. FreeBSD has a good view of the internals of these protocols and we can use system tools to understand how well they are performing. Application layer protocols such as HTTP can be a big factor in network performance, but analysis and tuning are tightly coupled to the application and out of scope for this article.
Two hosts are connected together by a series of links connected together through bridges, switches, and routers. The set of links that a packet passes through to get from one host to another is called a path. Paths should be assumed to be asymmetric (not the same in both directions) and expected to vary over time. Any measurement we take will give us values for that path, it might not apply well to other paths.
Establishing a Network Baseline
Before you can look at the performance of a system you first need to understand what ‘good’ performance for your host and network is. You need to have a set of baseline measurements from which you can determine if the network performance is lacking. The best way to do this is well in advance when machines are new and before they are seeing a deluge of traffic.
Measuring Delay
Delay is the most straight forward network characteristic to reason about, but the impact it can have on a connection’s performance can be unintuitive. The most common way to measure the delay between two hosts is to use the ‘ping’ tool. This is so common that most gamers will talk about their ‘ping’ time rather than delay at all. ping works by sending an ICMP Echo Request message and waiting for a matching ICMP Echo Reply message. The time between the request and the reply is the round-trip time.
Code:
[user@freebsd-laptop] $ ping -c 10 klarasystems.com
PING klarasystems.com (192.0.78.155): 56 data bytes
64 bytes from 192.0.78.155: icmp_seq=0 ttl=57 time=37.368 ms
64 bytes from 192.0.78.155: icmp_seq=1 ttl=57 time=140.856 ms
64 bytes from 192.0.78.155: icmp_seq=2 ttl=57 time=33.893 ms
64 bytes from 192.0.78.155: icmp_seq=3 ttl=57 time=34.486 ms
64 bytes from 192.0.78.155: icmp_seq=4 ttl=57 time=37.268 ms
64 bytes from 192.0.78.155: icmp_seq=5 ttl=57 time=35.898 ms
64 bytes from 192.0.78.155: icmp_seq=6 ttl=57 time=33.830 ms
64 bytes from 192.0.78.155: icmp_seq=7 ttl=57 time=35.521 ms
64 bytes from 192.0.78.155: icmp_seq=8 ttl=57 time=35.751 ms
64 bytes from 192.0.78.155: icmp_seq=9 ttl=57 time=34.804 ms
--- klarasystems.com ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 33.830/45.967/140.856/31.651 msWith the ping command we learn a lot of things about the state of the network right now, the configuration of the host and the nature of the delay component of the path. Firstly, we learn that the host is connected to the network, how many network ‘hops’ away it is, if there is any packet loss, that DNS works, that we can get out of the local network and that we have valid routing. We get all of this by the fact that the ping command worked and we received responses.
Secondly, we learn several things about the nature of the delay component of our network path to this host. We can look at the final column of the ping output (the one with time=) and get a sense for how consistent the delay is. Ping also provides two summary lines at the end of a session (when you use the -c count flag, or if you press ^C).
The first summary line tells you about packet loss. Loss is a normal feature of the network; excessive levels of loss are worth investigating.
The second summary line tells you about the nature of the variance of delay in the network. This is provided in the form of the min, average, max, and standard deviation in delay. The second packet in this transfer had a time of 140.865ms, this might be significant if it happens frequently, but we can’t tell in the space of 10 packets here.
We have to consider the delay for the path for each host as a point in time measurement. Different hosts will almost always be in different physical locations and so there will be a variation in the delay characteristics between them. The path between two hosts can change at any time and when we design network protocols one requirement is to always consider that the paths will be different in different directions. This commonly happens when there are ECMP load balancers in the ingress path of a network.
For good performance it is important that delay is consistent over time. Reliable protocols, such as TCP, use the smoothed round trip time (SRTT) to establish timers that govern loss recovery and reliability mechanisms. If there are frequent delay spikes or the variation is very large these timers might fire accidentally.
Performance can also be improved by moving hosts closer together to reduce the delay. At the core, this is one of the large benefits to using a Content Delivery Network. Any transactional workload will be governed by the speed of light and any network process that reacts or grows based on round trip times will be limited by how many trips can be achieved in a second. Delay is a big issue for doing large transfers across the planet, moving 100GB on a 1Gb link is harder between Frankfurt and Sydney than it is between Frankfurt and London because it takes longer to grow the window and longer to recover after a packet loss event.
Code:
[user@freebsd-laptop] $ ping -c 1000 klarasystems.com
PING klarasystems.com (192.0.78.155): 56 data bytes
64 bytes from 192.0.78.155: icmp_seq=0 ttl=57 time=37.878 ms
64 bytes from 192.0.78.155: icmp_seq=1 ttl=57 time=36.338 ms
...
64 bytes from 192.0.78.155: icmp_seq=996 ttl=57 time=471.185 ms
64 bytes from 192.0.78.155: icmp_seq=997 ttl=57 time=34.490 ms
64 bytes from 192.0.78.155: icmp_seq=998 ttl=57 time=33.977 ms
64 bytes from 192.0.78.155: icmp_seq=999 ttl=57 time=33.887 ms
--- klarasystems.com ping statistics ---
1000 packets transmitted, 999 packets received, 0.1% packet loss
round-trip min/avg/max/stddev = 33.469/37.932/471.185/19.235 msThe Internet is a best effort network, which means we expect packets to be lost and for delay to vary. The above trace was taken from my home network. Here we see a single packet getting dropped and one sample with a massive variation in delay. These two packets were probably victims to home WiFi.
Measuring Bandwidth
The actual speed a single connection can manage is influenced by a number of factors. The closest to the machine is the link speed of the network interface. For a wired connection such as Ethernet this will be something like 100Mbit if you are unlucky and more likely it will be 1Gb, 10Gbit or higher. Wired Network links are quite good at determining the speed actually available on the physical medium. Wireless links have a lot of factors that influence their speeds; you will probably see a link speed from 54Mbit with 80211g all the way up to a gigabit or more with WiFi 6.
The hosts link speed sets upper bound on the speed of any connection (ignoring aggregated links such as lagg(4)). A more likely factor will be the bandwidth made available via the service level agreement from the data center operator or ISP that offers the connection out to the internet. This quoted bandwidth will probably be a more realistic upper bound, with the real available value governed by network contention (how over-sold the link is) and congestion (how active the other users of the link are).
The capacity of the path is determined by the speed of the slowest link, how congested the link is, and how much buffering is on the link. Too little buffering and bursts of traffic will be dropping impacting performance and reducing throughput, but too much buffering and latency increases with traffic volume.
iperf3 (available as benchmarks/iperf3) is a standard, well-tested tool for measuring network performance. It has to be run as both the client and server and is able to report the throughput for TCP, UDP and SCTP connections and supports a host of options. There are other similar tools, it is important to use these consistently and be aware when you are comparing different versions of the software, as different versions can report different performance results.
iperf3 is also able to report results as JSON, which can be very helpful for integrating measurements into other tools.
To measure with iperf3 we need to run a server instance:
Code:
[user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
And connect from the client to perform measurements:
[user@freebsd-client] $ iperf3 -c freebsd-server.example.com
Connecting to host 10.0.4.2, port 5201
[ 5] local 10.0.4.1 port 35041 connected to 10.0.4.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 99.9 MBytes 838 Mbits/sec 331 431 KBytes
[ 5] 1.00-2.00 sec 193 MBytes 1.62 Gbits/sec 0 855 KBytes
[ 5] 2.00-3.00 sec 197 MBytes 1.65 Gbits/sec 16 174 KBytes
[ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec 0 787 KBytes
[ 5] 4.00-5.00 sec 196 MBytes 1.65 Gbits/sec 2 538 KBytes
[ 5] 5.00-6.00 sec 205 MBytes 1.72 Gbits/sec 0 934 KBytes
[ 5] 6.00-7.00 sec 217 MBytes 1.82 Gbits/sec 2 710 KBytes
[ 5] 7.00-8.00 sec 202 MBytes 1.69 Gbits/sec 0 1.02 MBytes
[ 5] 8.00-9.00 sec 166 MBytes 1.39 Gbits/sec 0 1.21 MBytes
[ 5] 9.00-10.00 sec 165 MBytes 1.38 Gbits/sec 0 1.37 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec 351 sender
[ 5] 0.00-10.01 sec 1.80 GBytes 1.54 Gbits/sec receiver
iperf Done.Did you know?
You can maximize the power of your FreeBSD infrastructure with our Support Subscription!
Check it out!The client will by default print the results for each second of transfer and perform the transfer for 10 seconds. When a client connects to the server, it will log the received rate for each second of transfer.
Code:
[user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.4.1, port 36348
[ 5] local 10.0.4.2 port 5201 connected to 10.0.4.1 port 35041
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 98.2 MBytes 823 Mbits/sec
[ 5] 1.00-2.00 sec 192 MBytes 1.60 Gbits/sec
[ 5] 2.00-3.00 sec 198 MBytes 1.67 Gbits/sec
[ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec
[ 5] 4.00-5.00 sec 196 MBytes 1.64 Gbits/sec
[ 5] 5.00-6.00 sec 205 MBytes 1.71 Gbits/sec
[ 5] 6.00-7.00 sec 218 MBytes 1.83 Gbits/sec
[ 5] 7.00-8.01 sec 201 MBytes 1.68 Gbits/sec
[ 5] 8.01-9.00 sec 166 MBytes 1.41 Gbits/sec
[ 5] 9.00-10.00 sec 165 MBytes 1.38 Gbits/sec
[ 5] 10.00-10.01 sec 935 KBytes 588 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 1.80 GBytes 1.54 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------By default, iperf3 sends traffic from the client to the server, this being usually backwards to what you want to measure. We can get the opposite behaviour using the -R flag. As iperf3 sends for 10 seconds, networks with a high delay component might not have long enough to get up to full speed. I find it better for comparisons to transfer a fixed amount of data using the -n flag (there is an iperf3 bug where quantities don’t work in reverse mode).
TCP carries HTTP and because it makes up the majority of traffic on the Internet it has seen a lot of optimization. TCP is more likely to have offload configured, and tests on real hardware are likely to have much higher TCP throughput than UDP.
UDP tests are also helpful to understand your network. iperf3 will be default only; try to send a 1Mbit/s with UDP and so you have to specify a target bit rate.
Code:
[user@freebsd-client] $ iperf3 -c 10.0.4.2 -u -b 1G
Connecting to host 10.0.4.2, port 5201
[ 5] local 10.0.4.1 port 16765 connected to 10.0.4.2 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 119 MBytes 1000 Mbits/sec 85586
[ 5] 1.00-2.00 sec 119 MBytes 1000 Mbits/sec 85587
[ 5] 2.00-3.00 sec 119 MBytes 1.00 Gbits/sec 85642
[ 5] 3.00-4.00 sec 119 MBytes 1000 Mbits/sec 85608
[ 5] 4.00-5.00 sec 119 MBytes 1.00 Gbits/sec 85626
[ 5] 5.00-6.00 sec 119 MBytes 1000 Mbits/sec 85590
[ 5] 6.00-7.00 sec 119 MBytes 1000 Mbits/sec 85607
[ 5] 7.00-8.00 sec 119 MBytes 1.00 Gbits/sec 85621
[ 5] 8.00-9.00 sec 119 MBytes 1000 Mbits/sec 85610
[ 5] 9.00-10.00 sec 119 MBytes 1000 Mbits/sec 85725
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.16 GBytes 1000 Mbits/sec 0.000 ms 0/856202 (0%) sender
[ 5] 0.00-10.10 sec 1.01 GBytes 859 Mbits/sec 0.003 ms 113542/856202 (13%) receiver
iperf Done.However, traffic on the parent interface that has a VLAN tag (traffic to 10.0.64.0/24 subnet) will show up in captures on that interface with the tag:
Code:
root@hostA # tcpdump -i vtnet1 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:59:12.596875 00:a0:98:67:03:ce (oui Unknown) > 00:a0:98:05:09:2c (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.1 > 10.0.64.2: ICMP echo request, id 33029, seq 0, length 64
19:59:12.600823 00:a0:98:05:09:2c (oui Unknown) > 00:a0:98:67:03:ce (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.64.2 > 10.0.64.1: ICMP echo reply, id 33029, seq 0, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernelWhen measuring UDP it is normally more helpful to note the rate at which that server can receive, rather than the rate at which the client can send. The host’s UDP stack will accept packets until all buffers are full and then silently drop packets, misrepresenting to the application how many packets are actually being sent each second. On the server side we will see the packets that actually make it across the network; you can see this in the below trace from a local virtual machine network.
Code:
[user@freebsd-server] $ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.4.1, port 38610
[ 5] local 10.0.4.2 port 5201 connected to 10.0.4.1 port 16765
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 94.9 MBytes 796 Mbits/sec 0.007 ms 9432/77555 (12%)
[ 5] 1.00-2.00 sec 103 MBytes 861 Mbits/sec 0.007 ms 11865/85575 (14%)
[ 5] 2.00-3.00 sec 104 MBytes 869 Mbits/sec 0.002 ms 11261/85638 (13%)
[ 5] 3.00-4.00 sec 104 MBytes 871 Mbits/sec 0.002 ms 11018/85613 (13%)
[ 5] 4.00-5.00 sec 111 MBytes 932 Mbits/sec 0.006 ms 5788/85597 (6.8%)
[ 5] 5.00-6.00 sec 101 MBytes 843 Mbits/sec 0.005 ms 13379/85585 (16%)
[ 5] 6.00-7.00 sec 100 MBytes 842 Mbits/sec 0.009 ms 13515/85605 (16%)
[ 5] 7.00-8.00 sec 104 MBytes 872 Mbits/sec 0.011 ms 10908/85581 (13%)
[ 5] 8.00-9.00 sec 103 MBytes 867 Mbits/sec 0.002 ms 11390/85618 (13%)
[ 5] 9.00-10.00 sec 100 MBytes 842 Mbits/sec 0.004 ms 13610/85702 (16%)
[ 5] 10.00-10.10 sec 9.41 MBytes 831 Mbits/sec 0.003 ms 1376/8133 (17%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.10 sec 1.01 GBytes 859 Mbits/sec 0.003 ms 113542/856202 (13%) receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------Because UDP is less likely to have effective offload support you will see that for the same transfer rate that TCP has much better CPU performance with iperf3. If you test on smaller devices (APU2 boards are a good test tool for this) you will observe that UDP only manages 400Mbit/s while TCP can saturate a Gigabit link. This happens because iperf3 becomes CPU limited for UDP, but TCP can offload a lot of the work to the network interface.
Passive Network Performance Information
The FreeBSD network stack gathers a lot of information as packets pass through, and these are exposed as statistic counters. You can access this information with netstat and the -s flag, and it is likely to give you much more than you can digest:
Code:
[user@freebsd-server] $ netstat -s
tcp:
26734809 packets sent
20912618 data packets (16879154564 bytes)
11369 data packets (2782329 bytes) retransmitted
90 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
5669526 ack-only packets (72899 delayed)
0 URG only packets
161 window probe packets
3609 window update packets
... 461 more lines ...
0 mbufs inserted during makespace
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams outputYou can grep for what you want in the netstat output and you can filter for specific protocols by using the -p flag.
Code:
[user@freebsd-server] $ netstat -s -p ip
ip:
35934055 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
31395385 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
38879648 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
31 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in headerDid you know?
We wrote about VNETs in “Virtualising Networks with FreeBSD VNET Jails“
Check it out!Statistic counters can be very helpful in monitoring activity, and detecting network failures and interface errors. In the ip, statistics above the total packets received and total packets sent tell us how active this host is. Any of the error condition counters such as ‘bad header checksums’, ‘bad options’ or ‘ip length > max ip size’ being non-zero or increasing can indicate misbehaving software and failing or badly configured network devices.
The IP counters for forwarding can be helpful to track and diagnose routing issues for which traces might be hard to capture. Netstat can also reveal if Path MTU Discovery is failing by showing both the incoming and outgoing fragmentation counters. While fragmentation is a requirement for the network to function, it is frequently an indicator of configuration issues somewhere on the path.
The FreeBSD tool systat makes it easier to track how network statistics change over time. systat offers a live display of network statistics and updates this information similar to how top behaves.
systat will provide information for network interfaces (via ifstat) and for network protocols (providing icmp, icmp6, ip, ip6, sctp and tcp).
systat has a command entry window which is accessed by typing a colon ‘:’ followed by a command string. This command window can be used to move between the different displays by typing their names. ‘:ifstat’ will take you to the ifstat display and ‘:tcp’ will take you to the TCP display.
systat -ifstat will provide current and peak transfer rates for all of the interfaces on a host marked as up, and will show a total amount of data transferred over the interface since systat was started. ifstat mode can show network information with different units with the scale command or as packets per second. It can also filter the interfaces shown, which is helpful on hosts with many virtual machines or jails.
Code:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average ||||
Interface Traffic Peak Total
lo0 in 0.000 KB/s 0.000 KB/s 51.187 KB
out 0.000 KB/s 0.000 KB/s 51.187 KB
vtnet0 in 204.499 MB/s 224.708 MB/s 4.985 GB
out 4.459 MB/s 4.900 MB/s 111.414 MBsystat can be helpful to determine how much traffic is passing through a host. With real traffic this can be used as a good comparison with the measurements taken with iperf3, as well as previous measurements of the host’s activity.
systat has six screens of network protocol output to show you. UDP information is shown alongside ip and ip6. All of these screens operate in one of four modes: rate, delta, since or absolute.
Code:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |||
TCP Connections TCP Packets
7752 connections initiated 297828 total packets sent
654 connections accepted 118740 - data
8186 connections established 529 - data (retransmit by dupack)
412 connections dropped 63 - data (retransmit by sack)
26 - in embryonic state 159860 - ack-only
8 - on retransmit timeout 0 - window probes
0 - by keepalive 3035 - window updates
0 - from listen queue 0 - urgent data only
15663 - control
0 - resends by PMTU discovery
TCP Timers 283757 total packets received
119689 potential rtt updates 176485 - in sequence
132848 - successful 2184 - completely duplicate
74552 delayed acks sent 2 - with some duplicate data
404 retransmit timeouts 1046 - out-of-order
0 persist timeouts 2777 - duplicate acks
0 keepalive probes 133702 - acks
0 - timeouts 0 - window probes
865 - window updates
5829 - bad checksum‘absolute’ mode shows the values of the counters as they are, the network protocol modes will start by showing you the absolute counters by default and then will show you rate of change on the first refresh. The other modes will show you how the counters are changing over time or since a point in time. The reference point for these can be set with the ‘reset’ command.
While you can watch the counters change, interpreting the information they provide requires understanding of the protocols and what the different counters represent.
In the TCP output above, the ‘TCP Connections’ and ‘TCP Timers’ sections tell us about connections and their status. Large numbers of connections in the ‘embryonic state’ could indicate a resource limit, a busy server or an attack. A large ‘listen queue’ might be a sign of a similar issue. Lots of connections in ‘keepalive’ might indicate a network or application protocol error. ‘persistent timeouts’ can show us that there is packet loss happening somewhere and we might want to look at performance on the path.
The ‘TCP Packets’ section tells us more raw information about the packets that make up our TCP connections. ‘sent’ and ‘received’ should be clear and represent the number of TCP segments sent and received. The ‘out-of-order’ and ‘duplicate acks’ fields can be signs of error. TCP is sensitive to reordering (hence out-of-order packets are an indication of a problem) and loss; loss typically appears as duplicate acks. This is a very quiet host on the Internet, with a long uptime and no performance issues I am aware of, so these numbers are not a concern.
Code:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |
ICMP Input ICMP Output
268946 total messages 226150 total messages
0 with bad code 8655 errors generated
9 with bad length 0 suppressed - original too short
5 with bad checksum 0 suppressed - original was ICMP
0 with insufficient data 217495 responses sent
0 suppressed - multicast echo
0 suppressed - multicast tstamp
Input Histogram Output Histogram
326 echo response 217485 echo response
217485 echo request 0 echo request
50469 destination unreachable 8655 destination unreachable
13 redirect 0 redirect
608 time-to-live exceeded 0 time-to-live exceeded
0 parameter problem 0 parameter problem
0 router advertisement 0 router solicitationConclusion
“Benchmarks are hard to get right, and network benchmarks are harder”
The Internet we have built is one of humanity’s most impressive achievements, but that achievement is built on a lot of complexity. This article has touched a little bit on how we define performance and the factors we control that influence the available performance. We have looked at how we can measure what is available and how we can get the system to report what it is managing to do. We have looked at some indicators of errors, but as happy families are all the same, all unhappy networks are unique in their own way.
FreeBSD does a very good job of pushing packets and offers a lot of facilities to help you improve what it is doing. This article has only touched on the surface of network performance analysis.
Meet the Author: Tom Jones
Tom Jones is an Internet Researcher and FreeBSD developer that works on improving the core protocols that drive the Internet. He is a
contributor to open standards in the IETF and is enthusiastic about using FreeBSD as a platform to experiment with new networking ideas as they progress towards standardisation.
Read more
Like this article? Share it!
- Share on Twitter
- Share on Linkedin
- Share via Email
- Share on Reddit
You might also want be interested in
Get more out of your FreeBSD networking
Networking is crucial to many companies. If you have a FreeBSD networking implementation or a driver you’re looking at, our team can help you further enable your efforts.
FreeBSD Support
FreeBSD Development
Continue reading...