DUP! ICMP redirect from localhost when using bridge0 - also bridge0 unicast flooding

Rudy · Dec 16, 2010

I have a bridge with 3 vlans all trunking off the same em1. When I ping across 2 of the vlans (7 and 715) I get a 'Redirect Host' from localhost. However, vlan714 pings just fine. What is up?

Code:

# ping 10.7.1.31
PING 10.7.1.31 (10.7.1.31): 56 data bytes
36 bytes from localhost (127.0.0.1): Redirect Host(New addr: 10.7.1.31)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 af97   0 0000  3f  01 b6e4 10.7.0.1  10.7.1.31 

64 bytes from 10.7.1.31: icmp_seq=0 ttl=64 time=286.316 ms
64 bytes from 10.7.1.31: icmp_seq=0 ttl=64 time=286.350 ms (DUP!)
36 bytes from localhost (127.0.0.1): Redirect Host(New addr: 10.7.1.31)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 af9f   0 0000  3f  01 b6dc 10.7.0.1  10.7.1.31 

64 bytes from 10.7.1.31: icmp_seq=1 ttl=64 time=198.031 ms
64 bytes from 10.7.1.31: icmp_seq=1 ttl=64 time=198.062 ms (DUP!)
^C
--- 10.7.1.31 ping statistics ---
2 packets transmitted, 2 packets received, +2 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 198.031/242.190/286.350/44.143 ms
# ping 10.7.1.5
PING 10.7.1.5 (10.7.1.5): 56 data bytes
64 bytes from 10.7.1.5: icmp_seq=0 ttl=64 time=5.160 ms
64 bytes from 10.7.1.5: icmp_seq=1 ttl=64 time=3.392 ms
64 bytes from 10.7.1.5: icmp_seq=2 ttl=64 time=6.250 ms
^C
--- 10.7.1.5 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.392/4.934/6.250/1.178 ms
# route show !$
route show 10.7.1.5
   route to: ap-rocket-05.wifi.monkeybrains.net
destination: 10.7.0.0
       mask: 255.255.0.0
  interface: bridge0
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 
# route show 10.7.1.31
   route to: ap-rocket-31.wifi.monkeybrains.net
destination: 10.7.0.0
       mask: 255.255.0.0
  interface: bridge0
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 
# uname -a
FreeBSD arana.monkeybrains.net 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Oct 20 15:55:41 PDT 2010     root@pulga.monkeybrains.net:/usr/obj/usr/src/sys/JEJEN  amd64
# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether ee:25:90:05:70:ff
	inet 10.7.0.1 netmask 0xffff0000 broadcast 10.7.255.255
	inet 10.7.208.1 netmask 0xfffffc00 broadcast 10.7.211.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vlan715 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 6 priority 128 path cost 20000
	member: vlan714 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 8 priority 128 path cost 20000
	member: vlan7 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 5 priority 128 path cost 2000000

SirDice · Dec 16, 2010

The ip address 10.7.0.1/16 overlaps with 10.7.208.1/22.

Can you post the complete output from ifconfig -a and netstat -rn?

Rudy · Dec 16, 2010

Having a subnet should not affect which device in the bridge is picked for transmission. Actually, I erased the 'real' gateways and the 10.7/16 is only for low bandwidth maintenance stuff.

Below is the output you requested (external ips munged into the 192 space). Has anyone ever seen bridges using the wrong dev or spitting out unicast floods? Also, is there a command to view the bridge/mac associations? ie, if ip 10.7.1.5 is on bridge0 member xxx0 vs xxx1, how do I view that?

Also, net.link.ether.inet.max_age: 1500

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
	ether 00:25:90:05:70:fe
	inet 192.168.15.4 netmask 0xfffffff8 broadcast 192.168.15.7
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
	ether 00:25:90:05:70:ff
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	inet 10.8.254.104 netmask 0xffffffff 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
vlan7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:25:90:05:70:ff
	inet 10.8.254.6 netmask 0xffffffe0 broadcast 10.8.254.31
	inet 192.168.15.57 netmask 0xfffffff8 broadcast 192.168.15.63
	inet 192.168.14.81 netmask 0xfffffff8 broadcast 192.168.14.87
	inet 192.168.14.65 netmask 0xfffffff8 broadcast 192.168.14.71
	inet 192.168.14.129 netmask 0xffffff80 broadcast 192.168.14.255
	inet 192.168.13.1 netmask 0xffffffe0 broadcast 192.168.13.31
	inet 192.168.13.65 netmask 0xffffffc0 broadcast 192.168.13.127
	inet 192.168.13.129 netmask 0xffffff80 broadcast 192.168.13.255
	inet 10.7.0.1 netmask 0xffff0000 broadcast 10.7.255.255
	inet 10.7.208.1 netmask 0xfffffc00 broadcast 10.7.211.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 7 parent interface: em1
vlan714: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:25:90:05:70:ff
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 714 parent interface: em1
vlan715: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 00:25:90:05:70:ff
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 715 parent interface: em1

and netstat.

Code:

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.15.1       UGS       148 101515657    em0
10.7.0.0/16        link#5             U           1   203789  vlan7
10.7.0.1           link#5             UHS         0      118    lo0
10.7.208.0/22      link#5             U           0     4340  vlan7
10.7.208.1         link#5             UHS         0        0    lo0
10.8.254.0/27      link#5             U           6   222736  vlan7
10.8.254.6         link#7             UHS         1        0    lo0
10.8.254.101       192.168.15.1       UGH1        0        0    em0
10.8.254.102       192.168.15.2       UGH1        0        0    em0
10.8.254.104       link#4             UH          0        0    lo0
10.77.0.0/24       192.168.15.2       UG1         0        0    em0
10.99.99.0/24      10.8.254.4         UG1         0      100  vlan7
38.99.18.136/29    192.168.15.1       UG1         0        0    em0
38.99.221.28/30    192.168.15.1       UG1         0       54    em0
64.215.30.152/30   192.168.15.2       UG1         0        0    em0
127.0.0.1          link#4             UH          0     2458    lo0
208.69.40.0/24     192.168.15.1       UG1         0    16591    em0
208.69.40.1        192.168.15.1       UGH1        0        0    em0
208.69.41.0/24     192.168.15.1       UG1         0      581    em0
208.69.42.0/27     192.168.15.1       UG1         0        0    em0
208.69.42.32/28    192.168.15.1       UG1         0        0    em0
208.69.42.48/28    192.168.15.1       UG1         0        0    em0
208.69.42.96/27    192.168.15.1       UG1         0        0    em0
208.69.42.128/25   192.168.15.1       UG1         0        0    em0
208.69.43.64/26    192.168.15.1       UG1         0        0    em0
208.69.43.192/27   192.168.15.1       UG1         0        0    em0
208.69.43.224/27   192.168.15.1       UG1         0        0    em0
192.168.13.0/27    link#5             U           0  2072220  vlan7
192.168.13.1       link#7             UHS         1        0    lo0
192.168.13.64/26   link#5             U           0   246943  vlan7
192.168.13.65      link#7             UHS         1        0    lo0
192.168.13.128/25  link#5             U           3 20167742  vlan7
192.168.13.129     link#7             UHS         1        0    lo0
192.168.14.64/29   link#5             U           0  1395771  vlan7
192.168.14.65      link#7             UHS         1        0    lo0
192.168.14.80/29   link#5             U           0    28239  vlan7
192.168.14.81      link#7             UHS         1        0    lo0
192.168.14.128/25  link#5             U           5 11115914  vlan7
192.168.14.129     link#7             UHS         1   389180    lo0
192.168.15.0/29    link#1             U           1     2212    em0
192.168.15.4       link#1             UHS         0        0    lo0
192.168.15.8/30    192.168.15.1       UG1         0        0    em0
192.168.15.12/30   192.168.15.1       UG1         0        0    em0
192.168.15.56/29   link#5             U           0    79253  vlan7
192.168.15.57      link#7             UHS         1        0    lo0
192.168.15.64/26   192.168.15.1       UG1         0       58    em0
192.168.15.128/27  192.168.15.1       UG1         0        0    em0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0
ff01:4::/32                       fe80::1%lo0                   U           lo0
ff02::%lo0/32                     fe80::1%lo0                   U           lo0

Rudy · Dec 16, 2010

Unicast flooding on bridge0

Looks like unicast flooding across all bridge0 members (vlan7/714/715) How do I stop that? I have 379 mac addresses in my arp table for that segment of the network.

Rudy · Dec 16, 2010

Doh typo

Man the TITLE should have been 'ICMP' not 'ISCP'. I was up late typing.

Can an admin fix that snafu?

SirDice · Dec 16, 2010

Rudy said:
Can an admin fix that snafu?

You can edit it yourself

Man, you have a complex setup. It's going to take me a while to figure out what's what

Few questions though, why 10 different subnets in vlan7 and none in 714 and 715?
And why connect them all on layer2? What's the reasoning behind this?
Perhaps I'm not getting the bigger picture but it's probably not the way I would connect things.

But I see no reason why 10.7.1.31 and 10.7.1.5 would be treated differently. Both go out on bridge0 -> vlan7 -> em1. The only thing I can think of is that it might be possible that the responses from 10.7.1.31 are returning via a different route compared to 10.7.1.5.

Did you take a peek at the traffic with tcpdump(1) yet?

Rudy · Dec 16, 2010

Ah, I found the advanced button so I could edit the title, thanks.

10.7.1.31 and 10.7.1.5 would be treated differently. Both go out on bridge0 -> vlan7 -> em1.

Actually, it goes like this bridge0 -> vlan714 -> em1 for 10.7.1.5 and bridge0 -> vlan715 -> em1 for 10.7.1.31, but I get no DUP! from bridge0 -> vlan7 -> em1 10.7.1.X on vlan7. Wierd.

Why do I have that setup? Because bridge0 supports PRIVATE

Client isolation. Actually, it is a little more complicated: I have a DHCP server and want to have a layer2 bridge so I can more efficiently pool my IPs across all segments of the network, however, I need to have the network segmented for many reasons... mostly to reduce unicast flooding on down stream switches, but also to provide client isolation. And all the smaller blocks of ips? They are for either [a] specific customers that need a /29 or for locking in a netblock to a specific vlan. I don't have dhcp on the smaller blocks, so I could directly assign them to the vlan members. Sometimes dhcp and static IPs live in the same vlan

Do people recommend mixing ips on bridges and on bridge members?

Oh, to get the system stable, I flattened all my switches to vlan7 and killed the bridge (hence all the IPs on vlan7 in the ifconfig below).

With the bridge with the 3 vlan members, the BSD box was unicast flooding every vlan with IP. Ouch. Way to bring down a network.

Long term goal: one bridge0 with about 50 vlans attached to it, FreeBSD sending packets properly to every vlan, DHCP listening on bridge0 serving up a /23 with one gateway IP.

SirDice · Dec 17, 2010

Rudy said:
Actually, it goes like this bridge0 -> vlan714 -> em1 for 10.7.1.5 and bridge0 -> vlan715 -> em1 for 10.7.1.31, but I get no DUP!

Look at your routing table, they both go out on vlan7. There are no IP addresses associated with vlan714 and vlan715 so nothing goes out on there.

Why do I have that setup? Because bridge0 supports PRIVATE Client isolation. Actually, it is a little more complicated: I have a DHCP server and want to have a layer2 bridge so I can more efficiently pool my IPs across all segments of the network, however, I need to have the network segmented for many reasons...

Network segmentation is good but keep the layer2 separated. If you need DHCP to cross a router use a DHCP relay agent (net/isc-dhcp3-relay for instance). On Cisco routers you'd use "ip-helper" for this.

Long term goal: one bridge0 with about 50 vlans attached to it, FreeBSD sending packets properly to every vlan, DHCP listening on bridge0 serving up a /23 with one gateway IP.

If you're going to bridge everything why use vlans at all? Keep the vlans seperated and use regular routing to connect everything. Set up DHCP relay agents where needed.

Rudy · Dec 17, 2010

SirDice said:
Look at your routing table, they both go out on vlan7. There are no IP addresses associated with vlan714 and vlan715 so nothing goes out on there.

Yeah, I had to abandon the bridge due to instability (customers calling). I didn't get a routing table copy-n-paste before I reverted. However, I did have all the IPs bridge0 (that you now see on vlan7) with the 3 vlans as members.

Network segmentation is good but keep the layer2 separated. If you need DHCP to cross a router use a DHCP relay agent (net/isc-dhcp3-relay for instance). On Cisco routers you'd use "ip-helper" for this.

I also need the same GW ip on the same vlans -- that doesn't help me.

If you're going to bridge everything why use vlans at all?

Unicast flooding -- in switches, not the router. I run a WISP and if a switch finds a packet that is not in it's mac-address-table, it turns it into a unicast flood. Sending that out all the antennas on a tower trashes the network throughput. 5Mbps of unicast = customers calling. I can set 'switchport unicast block' or the Cisco switches, but I have a few smaller crappier switches in remote spots that support vlans, but not tend to buckle and start unicast flooding over 30Mbps. The little NetGears (GS105E) are handy and support 4092 Mac Addresses, however, under load, the support more like 0 Mac Addresses.

Rudy · Dec 18, 2010

bridge can keep layer2 separate

SirDice said:
Network segmentation is good but keep the layer2 separated.

One note: The layer2 SHOULD be separate in my desired setup. vlan7 should never see vlan 714 as the bridge has PRIVATE set for the bridge members. Reference: http://www.freebsd.org/doc/handbook/network-bridging.html

My goal is to have one gw IP shared amongst vlans. In theory, a bridge should do this. Is anyone doing this in practice?

SirDice · Dec 20, 2010

Rudy said:
My goal is to have one gw IP shared amongst vlans.

You can't. For the simple reason that a gateway has to be inside the subnet. A gateway can never be outside of a subnet.

Rudy · Dec 22, 2010

SirDice said:
You can't. For the simple reason that a gateway has to be inside the subnet. A gateway can never be outside of a subnet.

You can, SirDice. I recommend you try an experiment to help build your networking skills. Setup two vlans, bridge them, then put an IP on the bridge.

If anyone else on this forum has done this and can help me, that would be great. Thanks for your attention, SirDice, but I am not even sure you have seen unicast flooding.

SirDice · Dec 23, 2010

Sure, like I have no experience after 20 years, maybe that's why my current employer trusts me with 15.000 routers and switches all over the globe. :OO

Good luck!

ccbs · Dec 5, 2011

Rudy said:
Ah, I found the advanced button so I could edit the title, thanks.

Long term goal: one bridge0 with about 50 vlans attached to it, FreeBSD sending packets properly to every vlan, DHCP listening on bridge0 serving up a /23 with one gateway IP.

Possible and easy to do.

Rudy · Dec 6, 2011

I'm using DHCP relay agent (helper-address)...
http://www.cisco.com/en/US/docs/ios/12_4t/ip_addr/configuration/guide/htdhcpre.html

As for the bridge0 spanning multiple vlans -- I am using it, but only for maintenance traffic. Outbound traffic is routed on layer 3 switches. I never did find a way to block unicast flooding via FreeBSD.

ccbs · Dec 12, 2011

On one router I have /22 network, each IP is in a different vlan, isc-dhcp on bridge0, different dummynet setup for each client. total bw ~450Mbit/s peak. Made some changes in if_bridge.c long ago, have to remember what was the case.

What was the unicast flood in your setup (torrent/ptp/virus)?

Rudy · Jan 28, 2012

Fix for unicast flooding

Many months ago, I had an issue with 'unicast flooding' on a bridge. A unicast flood is when a packet sent out of a bridge interface doesn't know which bridge member to use, so the packet is 'flooded' to all members. So much for the private flag.

Revisiting this issue, I reread ifconfig(8) and found the answer. if_bridge(4) could use some documentation on this matter.

The answer, is easy: up the bridge cache value like this:
[cmd=]# ifconfig bridge0 maxaddr 2000[/cmd]
If you think you are having 'flooding' issues, you can check how many addresses are cached for your bridge interface:
[cmd=]# ifconfig bridge0 addr | wc -l[/cmd]

Code:

If you have an untuned box and the number is pegged at '100' (the default cache size for bridges in 8.2), then you will get flooding.

What is the current maxaddr of your interface? Run ifconfig bridge0 and look for the maxaddr line!

Code:

# ifconfig bridge0
  bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:6e:81:55:e5:00
        inet 10.7.0.1 netmask 0xffff0000 broadcast 10.7.255.255
        inet 10.7.208.1 netmask 0xfffff800 broadcast 10.7.215.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp [B]maxaddr 2000[/B] [B][I]timeout 1200[/I][/B]
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan209 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 22 priority 128 path cost 20000
        member: vlan212 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 21 priority 128 path cost 20000
        member: vlan524 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000000
        etc, etc...

In the end, my friends, RTFM. A shame no one told me that months ago.

Oh, and the man page states:

Code:

     timeout seconds
             Set the timeout of address cache entries to seconds seconds.  If
             seconds is zero, then address cache entries will not be expired.
             The default is 240 seconds.

However, the bridge timeout is showing 1200 and I haven't touched it. I'm not sure if this value should match my arp timeout (300) or not... I'll have to play around with it. (ARP timeout = layer3 routing, bridge timeout = layer2 routing)

The command results of changing the maxaddr on a box with a saturated bridge cache table are shown the bandwidth graph attached.

DUP! ICMP redirect from localhost when using bridge0 - also bridge0 unicast flooding

Rudy

SirDice

Administrator

Rudy

Rudy

Attachments

Rudy

SirDice

Administrator

Rudy

SirDice

Administrator

Rudy

Rudy

SirDice

Administrator

Rudy

SirDice

Administrator

ccbs

Rudy

ccbs

Rudy

Attachments