Cannot route between two interfaces

  • Thread starter Thread starter Deleted member 79566
  • Start date Start date
D

Deleted member 79566

Guest
This is a continuation of this thread. I'm starting a new thread because it's been a while and because I've made some progress that simplifies the problem, plus I don't need fibs anymore.

I have a FreeBSD box acting as a dedicated firewall, with two interfaces: one connected to my modem (interface igb1) and one connected to my router (interface igb0). The firewall itself can access both the outside world and my local network, but it can't seem to route traffic between the two. (In the previous thread, the connection between the firewall and router was broken completely. I fixed that with dhcpd.)

Here's a general diagram of my setup. Excuse my GIMP:

network.png


Routing isn't completely broken, because I see outgoing traffic show up in both interfaces. I'm using Wireshark and ping to test. I can only ping the outside world from the firewall box. If I ping from inside my network, then I see the ping request go through both igb1 and igb0, but I never get a response.

Things I can do:
+ ping the firewall box (as 192.168.2.1) from my local network
+ ping and access services in my local network from the firewall box
+ ping and access the internet from the firewall box

Things I can't do:
+ ping or access the internet from my local network (and probably vice-versa, but I'll worry about port forwarding at a later point)

Note the subnet 192.168.2.0/24 is assigned by the firewall's dhcpd:
- 192.168.2.1 is the firewall (gateway)
- 192.168.2.10 is the router

Below is every config file and command that I thought could possibly be relevant:

cat /etc/rc.conf:
Code:
clear_tmp_enable="YES"
syslogd_flags="-ss"
hostname="myfirewall"
# (ignore nfe0 interface for now - I have it link down for testing)
ifconfig_nfe0="ether [[redacted custom MAC address]] DHCP"
ifconfig_nfe0_ipv6="inet6 accept_rtadv"
gateway_enable="YES"
defaultrouter="[[redacted my ISP's gateway]]"
ifconfig_igb1="ether [[redacted custom MAC address] DHCP"
ifconfig_igb1_ipv6="inet6 accept_rtadv"
ifconfig_igb0="inet 192.168.2.1/24"
ifconfig_igb0_ipv6="inet6 accept_rtadv"
moused_nondefault_enable="NO"
dumpdev="AUTO"
kld_list="i915kms"
# PF is disabled until I figure this out
pf_enable="NO"
pf_rules="/etc/pf.conf"
pflog_enable=yes
ipv6_gateway_enable=yes
dhcpd_enable="YES"
dhcpd_ifaces="igb0"

cat /etc/sysctl.conf:
Code:
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1

cat /usr/local/etc/dhcpd.conf:

Code:
option domain-name "informants.nsa.gov";

option subnet-mask 255.255.255.0;
default-lease-time 600;
max-lease-time 7200;

subnet 192.168.2.0 netmask 255.255.255.0 {
  range 192.168.2.10 192.168.2.100;
  option routers 192.168.2.1;
  option domain-name-servers 1.1.1.1, 8.8.8.8;
}

netstat -rn:
(NOTE: I have replaced all mentions of my public IP below with "11.22.33.44", which turns my ISP subnet to "11.22.33.0/23" and ISP gateway to "11.22.33.1".)
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            11.22.33.1         UGS        igb1
11.22.33.0/23      link#3             U          igb1
11.22.33.44        link#4             UHS         lo0
127.0.0.1          link#4             UH          lo0
192.168.2.0/24     link#2             U          igb0
192.168.2.1        link#4             UHS         lo0

Internet6:
[[redacted because idgaf about ipv6]]

ifconfig igb0:
Code:
igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether [[redacted MAC]]
    inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
    inet6 fe80::527c:6fff:fe52:ba28%igb0 prefixlen 64 scopeid 0x2
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

ifconfig igb1:
Code:
igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether [[redacted MAC]]
    hwaddr [[redacted MAC]]
    inet [[redacted public IP]] netmask 0xfffffe00 broadcast 255.255.255.255
    inet6 [[redacted ipv6]]%igb1 prefixlen 64 scopeid 0x3
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

edit: By the way, it's FreeBSD 14.1 x86_64
 
What is igb1 getting? Note that DHCP also receives a gateway address, that probably overwrites your set defaultrouter.

Also, what IP ranges are used on eth1, eth2 and eth3 on the router? Are those three different subnets? If that's the case you're going to need 3 static routes on the FreeBSD host, so it knows where to find those subnets. And it's a bad idea to have eth0 use a dynamic address, you're going to want to use static addresses here. Routing has to work both ways, packets need to know the way back to the system that initiated the connection. It doesn't magically remember the way back.

Internet6: [[redacted because idgaf about ipv6]]
Then why did you enable it everywhere? Including ipv6_gateway_enable?
 
What is igb1 getting? Note that DHCP also receives a gateway address, that probably overwrites your set defaultrouter.

Also, what IP ranges are used on eth1, eth2 and eth3 on the router? Are those three different subnets? If that's the case you're going to need 3 static routes on the FreeBSD host, so it knows where to find those subnets. And it's a bad idea to have eth0 use a dynamic address, you're going to want to use static addresses here. Routing has to work both ways, packets need to know the way back to the system that initiated the connection. It doesn't magically remember the way back.


Then why did you enable it everywhere? Including ipv6_gateway_enable?
The router with eth0/1/2/3 normally faces the Internet when the firewall isn't there, so it shouldn't need to be routed. It's a single subnet for all three computers in the local network: 192.168.1.0/24. I don't think my ISP has my internal subnet routed on their end... unless I'm missing something? The firewall machine (or the Internet for that matter) shouldn't know about 192.168.1.0/24 since that's a local subnet behind my router. I typically would access those machines using the router IP and a forwarded port, not via 192.168.1.0/24.

The `defaultrouter` there is the same gateway provided by my ISP.

Where do I have a dynamic address?

So in a nutshell there is a private network with a few clients and you want to reach the IPv4 Internet using your FreeBSD box as a gateway, right?

FreeBSD handbook Chapter 7

FreeBSD Handbook Chapter 30 to 34

It seems there is no chapter on pf and NAT, but Google should be able to help you here as well.
You think I made it this far without reading the manual section about networking...? ? Also chapter 30 is about dialup...?
 
The router with eth0/1/2/3 normally faces the Internet when the firewall isn't there, so it shouldn't need to be routed. It's a single subnet for all three computers in the local network: 192.168.1.0/24. I don't think my ISP has my internal subnet routed on their end... unless I'm missing something?
The FreeBSD host has to know it can find 192.168.1.0/24 behind the router's eth0 address. Or else the traffic will get send out to the default gateway.

The firewall machine (or the Internet for that matter) shouldn't know about 192.168.1.0/24 since that's a local subnet behind my router.
Yes, it does. The return packets have to be routed somewhere. The FreeBSD host only "knows" about the 192.168.2.0/24 network because it's directly connected to it. It has no idea where 192.168.1.0/24 is.

I typically would access those machines using the router IP and a forwarded port, not via 192.168.1.0/24.
So, the router does NAT? So you have NAT on the router, NAT on the FreeBSD host and I presume the cable modem does NAT too? Why?
 
The FreeBSD host has to know it can find 192.168.1.0/24 behind the router's eth0 address. Or else the traffic will get send out to the default gateway.
I'm a little confused now. I understand the concept of adding the "reverse" route within the context of a local network, but this is a router that typically faces the internet. When I remove the firewall from the equation, everything works fine without my ISP needing to add a route to 192.168.1.0/24, so how come I need to worry about that subnet when I replace my ISP with a firewall box?

Regardless, I added the route with the following command, but nothing seems to have changed:
Code:
route add 192.168.1.0/24 192.168.2.10

So, the router does NAT? So you have NAT on the router, NAT on the FreeBSD host and I presume the cable modem does NAT too? Why?
Why? ? You mean there's another way for me to insert the firewall into my network without doing any NAT...? I have no clue what that would even look like. Please do tell.

(I have no clue what the modem does; that's a black box from my point of view. It just has one ethernet input and one aux output.)
 
I'm a little confused now. I understand the concept of adding the "reverse" route within the context of a local network, but this is a router that typically faces the internet. When I remove the firewall from the equation, everything works fine without my ISP needing to add a route to 192.168.1.0/24, so how come I need to worry about that subnet when I replace my ISP with a firewall box?
So the router does NAT? Can you configure anything on that router? And by router I mean that left most box you've drawn in the schematic. I would just remove it, connect the 3 computers and the FreeBSD host's igb0 to a switch. There's zero added benefit of using a "router" here if the three interfaces (eth1-eth3) are not connected to different subnets.

Assuming the cable modem is in bridge mode, enable PF and NAT on igb1. Let the DHCP server on the FreeBSD deal with the computers. Maybe add unbound for some local caching DNS. Or use dnsmasq for an integrated DHCP/DNS solution on the FreeBSD host.

(I have no clue what the modem does; that's a black box from my point of view. It just has one ethernet input and one aux output.)
Is the cable modem in bridge or router mode? You can tell by the IP address that gets assigned to the FreeBSD host on igb1. If you get an RFC1918 (private ranges) address it's in router mode, if you get your 'external' Internet IP address it's in bridge mode.
 
On your firewall igb0 has an address on the subnet 192.168.2.0/24 and on igb1 an address on 11.22.33.1/23 and it is the firewall that doesn't route your traffic, right? You need pf enabled as it will do NAT.
# PF is disabled until I figure this out pf_enable="NO"
Run "tcpdump -n" on igb1 and check if any packets are getting through with src ip in 192.168.2.0/24 -- this is a private address space and anything out on the net wouldn't know how to get back to you. This is why you need NAT.
 
I have the same (actually more complex) environment.

Make sure these sysctls are set:

net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1

Make sure your firewall rules allow the traffic to egress and ingress (using stateful inspection or more complex individual rules). (I use ipfilter here.)

Also set up NAT, whether it be ipfw's natd, pf or ipfilter NAT.

Make sure your routing table includes all your networks. i.e. an example is worth a thousand words.

Code:
cwfw# netstat -nr4
Routing tables
 
Internet:
Destination        Gateway            Flags         Netif Expire
default            xx.xx.xx.xx        UGS             sk1
10.1.1.0/24        link#1             U               sk0
10.1.1.254         link#6             UHS             lo0
10.1.2.0/24        link#4             U              nfe0
10.1.2.254         link#6             UHS             lo0
64.59.160.40       xx.xx.xx.xx        UGHS            sk1
xx.xx.xx.xx/xx     link#2             U               sk1
xx.xx.xx.xx        link#6             UHS             lo0
127.0.0.1          link#6             UH              lo0
192.168.0.0/23     link#3             U              fxp0
192.168.0.1        link#6             UHS             lo0
192.168.0.2        link#6             UH              lo0
192.168.2.0/24     link#5             U              nfe1
192.168.2.254      link#6             UHS             lo0
192.168.9.0/29     link#8             U           epair0a
192.168.9.1        link#6             UHS             lo0
cwfw#
 
Make sure these sysctls are set:

net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1
I forgot to mention I did set those.

Make sure your firewall rules allow the traffic to egress and ingress (using stateful inspection or more complex individual rules). (I use ipfilter here.)
I disabled the firewall (pf) with the intention of figuring that part out once I fix the routing. However it's looking like maybe I needed it the whole time:

You need pf enabled as it will do NAT.
Also set up NAT, whether it be ipfw's natd, pf or ipfilter NAT.
I had not realized I needed an extra program for NAT. I thought NAT was just configuring the network routes and setting the forwarding sysctls. ?‍♂️ I guess that could be the part I'm missing. I already configured pf so I'm trying my luck with that first. I've enabled pf and rebooted the system to play around with NAT, but I didn't figure it out. The following line was the best I got:
Code:
nat on igb1 from {igb0:network} to any -> igb1
I tried various iterations of that to no avail. What else could I be missing?

Run "tcpdump -n" on igb1 and check if any packets are getting through with src ip in 192.168.2.0/24 -- this is a private address space and anything out on the net wouldn't know how to get back to you. This is why you need NAT.
Indeed, everything shows up with source IP 192.168.2.10. ?‍♂️

So the router does NAT? Can you configure anything on that router? And by router I mean that left most box you've drawn in the schematic. I would just remove it, connect the 3 computers and the FreeBSD host's igb0 to a switch. There's zero added benefit of using a "router" here if the three interfaces (eth1-eth3) are not connected to different subnets.

Assuming the cable modem is in bridge mode, enable PF and NAT on igb1. Let the DHCP server on the FreeBSD deal with the computers. Maybe add unbound for some local caching DNS. Or use dnsmasq for an integrated DHCP/DNS solution on the FreeBSD host.


Is the cable modem in bridge or router mode? You can tell by the IP address that gets assigned to the FreeBSD host on igb1. If you get an RFC1918 (private ranges) address it's in router mode, if you get your 'external' Internet IP address it's in bridge mode.
The router is pretty configurable. There are a few reasons to keep it around:
  • redundancy: two firewalls are better than one, specially two different firewalls on two different types of systems
  • compartmentalization: the firewall box is just a firewall, a smaller attack surface
  • security: FreeBSD is much easier (and less stress inducing) to keep up to date, which is convenient for the front door
  • resources: I'm trying to get away with a dual-core Athlon on the firewall, so the less work it has to do the better (although an upgrade to quad core is available if it turns out it can't even do the basic routing and firewalling I'm expecting of it without breaking a sweat)
 
You need something like the following in your /etc/pf.conf:

int_if="igb0"
ext_if="igb1"
internal_net="192.168.2.0/24"
icmp_types="{ echoreq unreach }"
set skip on lo0
set timeout { interval 1000 }
nat on $ext_if inet from ! ($ext_if) to any -> ($ext_if)
pass from { lo0, $internal_net } to any keep state
pass out on $ext_if proto { tcp, udp } all keep state
pass out inet proto icmp all icmp-type $icmp_types
pass in on $int_if from any to any tag INTNET
pass out quick on $ext_if tagged INTNET


If you want access from the net via ssh you'd need to add another rule. Something like

pass in on $ext_if proto tcp from any to $ext_if port 22 keep state


Note: this is just a hint. I am not an expert. If this doesn't work, you're on your own!
 
Back
Top