Solved Ping response not going out VLAN interface?

arader

arader

I'm trying to create two VLANs on my network, one for WAN destined traffic and another for internal infrastructure (DNS, DHCP, etc)

WAN VLAN: 254
INFRA VLAN: 253

My firewall machine (FreeBSD 12.0-p3) will be serving as both the WAN firewall (VLAN 254) as well as the DNS/DHCP server (VLAN 253). My intent is to have the VLAN 253 services inside a jail. Then, my switch has a third VLAN (240) where my computers all sit.

Setup:
  1. firewall has a single 10G connection to a Brocade ICX-6450 switch
  2. firewall has two vlan interfaces: cxgbe0.254 and cxgbe0.253
  3. firewall has two FIBs, FIB 0 and FIB 1
  4. My switch has three vlans: 254, 253, and 240
  5. FIB 0 has the WAN route (default through my ISP) as well as a route for VLAN 240 back through my switch
  6. FIB 1 has a default route of the switch's VLAN 253 address
  7. All machines on the 240 VLAN have a default route of the switch
Here's my problem. When a device on the 240 VLAN pings my firewall's VLAN 253 address, the response from my firewall doesn't appear to be sent, or at least, not sent over the VLAN 253 interface.

Details:
Code: 
   ____________________________                   ___________________________
  |  SWITCH                    |                 |  HOST A                   |
  |  VLAN 254 (172.16.254.254) |_________________|                           |
  |  VLAN 253 (172.16.253.254) |_________________|  VLAN 240 (172.16.240.10) |
  |  VLAN 240 (172.16.240.254) |                 |___________________________|
  |____________________________|
     | 10G Trunk |
   __|           |________________________
  |  FIREWALL                             |
  |  cxgbe0.254 VLAN 254 (172.16.254.1)   |
  |  cxgbe0.253 VLAN 253 (172.16.253.1)   |
  |_______________________________________|
FIREWALL:
Code: 
>ifconfig cxgbe0.254
cxgbe0.254: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:11:df:40
        inet 172.16.254.1 netmask 0xffffff00 broadcast 172.16.254.255
        inet6 fe80::207:43ff:fe11:df40%cxgbe0.254 prefixlen 64 scopeid 0x18
        groups: vlan
        vlan: 254 vlanpcp: 0 parent interface: cxgbe0
        media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Code: 
> ifconfig cxgbe0.253
cxgbe0.253: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:11:df:40
        inet 172.16.253.1 netmask 0xffffff00 broadcast 172.16.253.255
        inet6 fe80::207:43ff:fe11:df40%cxgbe0.253 prefixlen 64 scopeid 0xc
        groups: vlan
        vlan: 253 vlanpcp: 0 parent interface: cxgbe0
        fib: 1
        media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Notice that cfxgbe0.253 is in fib #1

Code: 
> netstat -rn4 -F 0
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS        igb0
127.0.0.1          link#9             UH          lo0
172.16.240.0/24    172.16.254.254     UGS    cxgbe0.2
172.16.253.0/24    link#12            U      cxgbe0.2
172.16.254.0/24    link#24            U      cxgbe0.2
172.16.254.1       link#24            UHS         lo0
192.168.0.0/24     link#5             U          igb0
192.168.0.8        link#5             UHS         lo0
Code: 
> netstat -rn4 -F 1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.253.254     UGS    cxgbe0.2
127.0.0.1          link#9             UH          lo0
172.16.253.0/24    link#12            U      cxgbe0.2
172.16.253.1       link#12            UHS         lo0
172.16.254.0/24    link#24            U      cxgbe0.2
192.168.0.0/24     link#5             U          igb0

I set up tcpdump on the firewall's cxgbe0.253 interface, and when I ping 172.16.253.1 from 172.16.240.10 I see the incoming ping requests, but no outgoing responses. The ping requests keep timing out on 172.16.240.10. In my pf.conf I have pass in inet proto icmp so it shouldn't be blocking any packets.

What's weird, is that the I can query dns entries from 172.16.240.1 using the dns server on 172.16.253.1 so it's not like the connection is totally not working. Perhaps even weirder, that when the dns server sends it's reponse, it's being sent over the VLAN 254 interface!?

any thoughts on what's going on here? I'm not using VNET or anything weird yet, I didn't think I'd have to for this setup.
 
ok I solved it, after searching around it looks like I was missing the following:
Code: 
net.add_addr_allfibs=0
in /etc/sysctl.conf. Adding that fixed both issues, now VLAN 253 traffic only traverses cxgbe0.253
 
You must log in or register to reply here.
Back
Top