if_bridge stops working after a while

R

rm_sys_dev

I'm running a 8.1-STABLE box which runs a transparent squid server. I have pf rdr'ing to squid successfully and bridging works for a while until it stops working. I haven't seen any specific trigger as to when it stops working (time or traffic). Essentially, when it stops working, I can no longer ping from one side of the network to the other side of the bridge or pass any traffic between both NIC, but from within FreeBSD I can ping both sides. Rebooting has been to only fix so far.

LAN -> ue0 -> bridge0 [pf to squid for port 80] -> em0 -> WAN


Everything is kept simple:
ifconfig:
Code: 
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
       options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
       ether 00:25:64:cb:b7:8e
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
pfsync0: flags=0<> metric 0 mtu 1460
       syncpeer: 224.0.0.240 maxupd: 128
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
       options=3<RXCSUM,TXCSUM>
       inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
       inet6 ::1 prefixlen 128
       inet 127.0.0.1 netmask 0xff000000
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
pflog0: flags=0<> metric 0 mtu 33152
ue0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
       options=80000<LINKSTATE>
       ether 00:25:4b:fd:c8:2c
       media: Ethernet autoselect (100baseTX <full-duplex>)
       status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       ether 5e:ad:20:eb:55:ed
       inet 192.168.3.254 netmask 0xffffff00 broadcast 192.168.3.255
       id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
       maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
       root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
       member: ue0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
               ifmaxaddr 0 port 6 priority 128 path cost 2000000
       member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
               ifmaxaddr 0 port 1 priority 128 path cost 2000000

rc.conf:
Code: 
hostname="squidbox.nothing.org"

cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 addm ue0 up"
ifconfig_bridge0_alias0="inet 192.168.3.254 netmask 255.255.255.0"
ifconfig_em0="up"
ifconfig_ue0="up"
defaultrouter="192.168.3.1"
pf_enable="YES"
sshd_enable="YES"
squid_enable="YES"
apache22_enable="YES"

pf.conf:
Code: 
int_if="ue0"
ext_if="em0"

rdr on $int_if inet proto tcp from any to any port www -> 127.0.0.1 port 3128
pass in quick on $int_if route-to lo0 inet proto tcp from any to
127.0.0.1 port 3128 keep state

Any suggestions?
 
Rules look ok. Do you have a skip rule on localhost? Do you have any block log rules and do you check tcpdump(1) on all interfaces, including pflog0 when things go wrong? It would be nice to know where the packet flow stops and/or where replies do not take place.
 
No other rules than what you see in my pf.conf file. I posted the complete pf.conf file.

Here is a tcpdump -v -i ue0 of a failed bridge:

Code: 
17:02:25.718356 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.98, length 46
17:02:26.051107 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:26.429115 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.52, length 46
17:02:26.577857 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:26.747982 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.98, length 46
17:02:27.522355 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.64, length 46
17:02:27.577855 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:27.753107 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.110, length 46
17:02:28.130607 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.64, length 46
17:02:28.366111 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.119, length 46
17:02:28.595357 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.52, length 46

and

tcpdump -v -i em0

Code: 
17:02:25.718356 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.98, length 46
17:02:26.051107 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:26.429115 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.52, length 46
17:02:26.577857 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:26.747982 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.98, length 46
17:02:27.522355 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.64, length 46
17:02:27.577855 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.87, length 46
17:02:27.753107 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.110, length 46
17:02:28.130607 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.64, length 46
17:02:28.366111 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.119, length 46
17:02:28.595357 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.52, length 46

Any help appreciated.
 
Again my bad, em0:

Code: 
17:02:40.567111 IP (tos 0x0, ttl 64, id 36994, offset 0, flags [none],
proto UDP (17), length 71)
   192.168.3.254.52959 > 192.168.3.1.domain: 29263+ PTR?
98.3.168.192.in-addr.arpa. (43)
17:02:40.568090 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 71)
   192.168.3.1.domain > 192.168.3.254.52959: 29263 NXDomain* 0/0/0 (43)
17:02:40.568183 IP (tos 0x0, ttl 64, id 36995, offset 0, flags [none],
proto UDP (17), length 72)
   192.168.3.254.50123 > 192.168.3.1.domain: 29264+ PTR?
255.3.168.192.in-addr.arpa. (44)
17:02:40.569150 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 72)
   192.168.3.1.domain > 192.168.3.254.50123: 29264 NXDomain* 0/0/0 (44)
17:02:40.569205 IP (tos 0x0, ttl 64, id 36996, offset 0, flags [none],
proto UDP (17), length 71)
   192.168.3.254.54285 > 192.168.3.1.domain: 29265+ PTR?
95.3.168.192.in-addr.arpa. (43)
17:02:40.570181 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 71)
   192.168.3.1.domain > 192.168.3.254.54285: 29265 NXDomain* 0/0/0 (43)
17:02:40.570239 IP (tos 0x0, ttl 64, id 36997, offset 0, flags [none],
proto UDP (17), length 71)
   192.168.3.254.31507 > 192.168.3.1.domain: 29266+ PTR?
68.3.168.192.in-addr.arpa. (43)
17:02:40.571213 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 71)
   192.168.3.1.domain > 192.168.3.254.31507: 29266 NXDomain* 0/0/0 (43)
17:02:40.571270 IP (tos 0x0, ttl 64, id 36998, offset 0, flags [none],
proto UDP (17), length 71)
   192.168.3.254.58980 > 192.168.3.1.domain: 29267+ PTR?
84.3.168.192.in-addr.arpa. (43)
17:02:40.572249 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 71)
   192.168.3.1.domain > 192.168.3.254.58980: 29267 NXDomain* 0/0/0 (43)
17:02:40.652124 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.52, length 46
17:02:40.652454 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.3.1
is-at 00:1f:33:1f:96:b5 (oui Unknown), length 46
17:02:40.663368 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.98, length 46
17:02:40.663694 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.3.1
is-at 00:1f:33:1f:96:b5 (oui Unknown), length 46
17:02:41.180082 IP (tos 0x0, ttl 128, id 20958, offset 0, flags
[none], proto UDP (17), length 78)
   192.168.3.95.netbios-ns > 192.168.3.255.netbios-ns: NBT UDP
PACKET(137): QUERY; REQUEST; BROADCAST
17:02:41.206743 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.84, length 46
17:02:41.207066 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.3.1
is-at 00:1f:33:1f:96:b5 (oui Unknown), length 46
17:02:41.236368 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.85, length 46
17:02:41.236687 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.3.1
is-at 00:1f:33:1f:96:b5 (oui Unknown), length 46
17:02:41.323115 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
192.168.3.1 tell 192.168.3.68, length 46
 
Why do you use a bridge for ? Simply put em0 and ue0 on different network.
 
Because em0 and ue0 is on the same network and I can't change that. Basically it is like this:

Internet provider -> Firewall/DHCP -> Squid -> Clients.

And squid has to run transparent to the client.
 
At least add the following to /etc/pf.conf:

Code: 
set skip on bridge0
set skip on lo0
 
Here's the update:

set skip on bridge0 stops squid from working correctly (clients will not get their HTTP requests)
set skip on lo0 doesn't seem to do anything to solve my problem, but squid works (until the bridge dies magically).

So I'm still stuck at the same problem... can't figure out why it stops after a while.
 
No, I was under the impression that is was not required for bridging. Would not having the option cause it to work and then stop at one point?

Thanks for trying to help me BTW, greatly appreciated.
 
I do think you need it. It looks like the bridge is now separating the networks (which is why ARP dies on both sides). Squid itself doesn't need forwarding (it acts as a 'mediator' due to your rdr and route-to rules), which may make it look like everything's working, but stuff that needs to traverse the bridge (DNS, maybe DHCP, stuff like that) may time-out. Not sure, the 'works for a while' situation is unclear, but try. You may, of course, need to tighten your firewall rules when you 'open the gates'.
 
Thank you for your response. I've added
Code: 
gateway_enable="YES"
into my /etc/rc.conf, now I can confirm that net.inet.ip.forwarding=1. I'll let it run a while to see if that fixed the issue.

Thanks again!
 
Sorry for the late answer, I was away for a while.

So even with IP Forwarding enabled, the issue is still there. I'm running out of options right now. I ordered a pci NIC to replace that USB (ue0) one, but I'm working in a 3rd world country and shipping takes a long time.

Edit:
Seems as though it may be related to this problem: http://forums.freebsd.org/showthread.php?t=8661.
 
I know that bridges are notoriously fidgety when one of the bridge interfaces is a wireless one. The same may indeed hold true for USB interfaces.
 
Alright,

So I made a simple script and added that to my /etc/crontab to run every 15 minutes. It's not the prettiest thing to do but until I get my new NIC, it's the only solution I found:

ue0.reset.sh:
Code: 
#!/bin/sh
usbconfig -u 7 -a 2 reset
sleep 3
ifconfig bridge0 addm ue0 up

1. So the usbconfig line resets my unit 7, address 2 device (which is my USB ethernet adapter)
2. Wait 3 seconds
3. Add ue0 to the bridge interface and bring it up again (after a reset, bridge0 looses that member)

Posted here in case someone might need it. Still, someone should look at PR 140883, there's obviously a problem in the driver.
 
If you need *real* help, report your problem on freebsd-net and freebsd-usb. I don't know if Hans Peter Selasky or Pyon YongHyeon are lurking that much the FreeBSD forums.
 
You must log in or register to reply here.
Back
Top