Need help analysing why ping reports (DUP!) replies from a FreeBSD 11.1 machine.

petur · May 12, 2018

I've spent a few hours in the near past to analyse a problem in which pinging a server results in duplicated packages if the package size is specified to be less than 1473.
I get the same reply when performing the ping using different clients. I also connected client and the server using a different switch, the DUPs persist.

The server is running a few jails and a single bhyve vm. Pinging the jails or the vm does not result in DUP replies.

I need help figuring out what why the server is sending out duplicates.

The following listing shows a default ping with two duplicate replies.

Code:

petur:~ petur$ ping server
PING server (192.168.2.150): 56 data bytes
64 bytes from 192.168.2.150: icmp_seq=0 ttl=64 time=1.943 ms
64 bytes from 192.168.2.150: icmp_seq=0 ttl=255 time=2.673 ms (DUP!)
64 bytes from 192.168.2.150: icmp_seq=1 ttl=64 time=2.169 ms
64 bytes from 192.168.2.150: icmp_seq=1 ttl=255 time=2.447 ms (DUP!)
64 bytes from 192.168.2.150: icmp_seq=2 ttl=64 time=1.434 ms

The following listing shows a partial result of pinging with package size 1472, a single ping with a duplicate reply.

Code:

PING server (192.168.2.150): 1472 data bytes
1480 bytes from 192.168.2.150: icmp_seq=0 ttl=64 time=2.123 ms
1480 bytes from 192.168.2.150: icmp_seq=0 ttl=255 time=2.361 ms (DUP!)

The following listing shows the use of package size 1473. No duplicates are returned if the package size is set to an equal or larger value than 1473.

Code:

PING server (192.168.2.150): 1473 data bytes
1481 bytes from 192.168.2.150: icmp_seq=0 ttl=64 time=2.399 ms
1481 bytes from 192.168.2.150: icmp_seq=1 ttl=64 time=2.552 ms

The following listing shows the result of executing ifconfig.

Code:

server# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=40098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO>
    ether 6c:0b:84:a9:ba:cd
    hwaddr 6c:0b:84:a9:ba:cd
    inet 192.168.2.150 netmask 0xffffff00 broadcast 192.168.2.255
    inet 192.168.2.153 netmask 0xffffffff broadcast 192.168.2.153
    inet 192.168.2.151 netmask 0xffffffff broadcast 192.168.2.151
    inet 192.168.2.156 netmask 0xffffffff broadcast 192.168.2.156
    inet 192.168.2.152 netmask 0xffffffff broadcast 192.168.2.152
    inet 192.168.2.158 netmask 0xffffffff broadcast 192.168.2.158
    inet 192.168.2.157 netmask 0xffffffff broadcast 192.168.2.157
    inet 192.168.2.159 netmask 0xffffffff broadcast 192.168.2.159
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.3.1 netmask 0xffffffff
    inet 127.0.1.1 netmask 0xffffffff
    inet 127.0.6.1 netmask 0xffffffff
    inet 127.0.2.1 netmask 0xffffffff
    inet 127.0.8.1 netmask 0xffffffff
    inet 127.0.7.1 netmask 0xffffffff
    inet 127.0.9.1 netmask 0xffffffff
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-public
    ether 02:29:95:a2:a2:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-host4-0-public
    options=80000<LINKSTATE>
    ether 00:bd:b0:05:f7:00
    hwaddr 00:bd:b0:05:f7:00
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 4287

The following listing shows the output of vm switch info

Code:

server# vm switch info
------------------------
Virtual Switch: public
------------------------
  type: auto
  ident: bridge0
  vlan: -
  nat: -
  physical-ports: em0
  bytes-in: 640685796800 (596.685G)
  bytes-out: 658414895272 (613.196G)

  virtual-port
    device: tap0
    vm: host4

The following listing shows the output of vm info.

Code:

server# vm info
------------------------
Virtual Machine: host4
------------------------
  state: running (4287)
  datastore: default
  loader: grub
  uuid: f7b16519-32c4-11e8-a944-6c0b84a9bacd
  uefi: no
  cpu: 4
  memory: 4196M
  memory-resident: 4309463040 (4.013G)

  console-ports
    com1: /dev/nmdm0B

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:06:dd:24
    fixed-device: -
    active-device: tap0
    desc: vmnet-host4-0-public
    mtu: 1500
    bridge: bridge0
    bytes-in: 623346114398 (580.536G)
    bytes-out: 45182855038 (42.079G)

  virtual-disk
    number: 0
    device-type: sparse-zvol
    emulation: virtio-blk
    options: -
    system-path: /dev/zvol/zroot/vm/host4/disk0
    bytes-size: 21474836480 (20.000G)
    bytes-used: 3713544192 (3.458G)

The following listing shows the result of executing jls.

Code:

server# jls
   JID  IP Address      Hostname                      Path
     1  127.0.3.1       host3                         /usr/jails/host3
     2  127.0.1.1       host1                          /usr/jails/host1
     3  127.0.6.1       host6                      /usr/jails/host6
     4  127.0.2.1       host2                         /usr/jails/host2
   155  127.0.8.1       host8                           /usr/jails/host8
   156  127.0.7.1       host7                         /usr/jails/host7
   185  127.0.9.1       host9                           /usr/jails/host9

The above listing relates to the listing for ifconfig -v by the following rule: A jail on the lo1 IP 127.0.x.1 has the em0 IP 129.168.2.15x.

The following listing shows the result of executing netstat -r.

Code:

server# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            USG                UGS         em0
localhost          link#2             UH          lo0
127.0.1.1          link#3             UH          lo1
127.0.2.1          link#3             UH          lo1
127.0.3.1          link#3             UH          lo1
127.0.6.1          link#3             UH          lo1
127.0.7.1          link#3             UH          lo1
127.0.8.1          link#3             UH          lo1
127.0.9.1          link#3             UH          lo1
192.168.2.0/24     link#1             U           em0
server               link#1             UHS         lo0
host1               link#1             UHS         lo0
192.168.2.151/32   link#1             U           em0
host2             link#1             UHS         lo0
192.168.2.152/32   link#1             U           em0
host3             link#1             UHS         lo0
192.168.2.153/32   link#1             U           em0
host6           link#1             UHS         lo0
192.168.2.156/32   link#1             U           em0
host7              link#1             UHS         lo0
192.168.2.157/32   link#1             U           em0
host8                link#1             UHS         lo0
192.168.2.158/32   link#1             U           em0
host9              link#1             UHS         lo0
192.168.2.159/32   link#1             U           em0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              localhost          UGRS        lo0
localhost          link#2             UH          lo0
::ffff:0.0.0.0/96  localhost          UGRS        lo0
fe80::/10          localhost          UGRS        lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
ff02::/16          localhost          UGRS        lo0

Lamia · May 12, 2018

Please include your firewall config - pf.conf or iptables. I need check that all is fine in there.

petur · May 13, 2018

Lamia said:
Please include your firewall config - pf.conf or iptables. I need check that all is fine in there.

There is no firewall running on the machine.

Max212 · May 13, 2018

is server connected to network via one or two or more network cables?

petur · May 14, 2018

Max212 said:
is server connected to network via one or two or more network cables?

One, as can be seen by the provided output of ifconfig.

VladiBG · May 14, 2018

20 byte (IP header) - 8 byte (ICMP header)
The MTU size of your interface is 1500 so the packet is fragmented when you using more than 1472 bytes of payload data.

The duplication occurs when:

a) echo requests are getting duplicated in network and end host is responding to each
b) end device receives correct echo request, and the response is getting duplicated somewhere in the network
c) two devices respond to the same ping request (like in broadcast ping)

You can use tcpdump to see what happen (watch for the MAC layer address)

tcpdump -ennvvi em0 icmp

edit:
also check your arp table.

PacketMan · May 18, 2018

I know of a case where a network switch was the culprit. It was cloning request or the response. It was reloaded and the issue cleared.

Need help analysing why ping reports (DUP!) replies from a FreeBSD 11.1 machine.

petur

Lamia

petur

Max212

petur

VladiBG

PacketMan