Solved Router(s) not forwarding packets to default gateway(s)

I have a FreeBSD router that doesn't seem to be forwarding packets to its default gateway, and I can't see why that might be.

Here's a diagram of the network:
diagram.jpg

I would like to get traffic from client -> router1 -> gateway -> internet.

client is able to ping router1, and router1 is able to ping gateway as well as 8.8.8.8 . client can't ping gateway or 8.8.8.8 . My powers of deduction tell me the issue lies somewhere in the configuration of router1. client:~ # traceroute 8.8.8.8 shows the first hop to router1 at 192.168.1.1 and then stalls out. The situation is similar for server, but I imagine that whatever I have to fix for one network branch will tell me how to fix the other.

Client /etc/rc.conf:
Code:
hostname="client"

growfs_enable="YES"

sshd_enable="YES"

# clientNet
defaultrouter="192.168.1.1"
ifconfig_em0="inet 192.168.1.2 netmask 255.255.255.0"

Client netstat -rn:
Code:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS         em0
127.0.0.1          link#3             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.2        link#1             UHS         lo0
192.168.101.0/24   link#2             U           em1
192.168.101.1      link#2             UHS         lo0

Router 1 /etc/rc.conf:
Code:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em1
127.0.0.1          link#4             UH          lo0
192.168.0.0/24     link#2             U           em1
192.168.0.2        link#2             UHS         lo0
192.168.1.0/24     link#3             U           em2
192.168.1.1        link#3             UHS         lo0
192.168.2.0/24     192.168.0.3        UGS         em1
192.168.99.0/24    link#1             U           em0
192.168.99.1       link#1             UHS         lo0

Router 1 netstat -rn:
Code:
hostname="router-1"

growfs_enable="YES"

gateway_enable="YES"

sshd_enable="YES"

# vboxnet0
ifconfig_em0="inet 192.168.99.1 netmask 255.255.255.0"

# routerNet
defaultrouter="192.168.0.1"
ifconfig_em1="inet 192.168.0.2 netmask 255.255.255.0"

# clientNet
ifconfig_em2="inet 192.168.1.1 netmask 255.255.255.0"

# static routes
static_routes="routerNet"
route_routerNet="-net 192.168.2.0/24 192.168.0.3"

Gateway /etc/rc.conf:
Code:
hostname="gateway"

growfs_enable="YES"

routed_enable="YES"

sshd_enable="YES"

# Enable and configure gateway
gateway_enable="YES"
pf_enable="YES"  
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
pflogd_enable="YES"
pfsync_enable="YES"

# WAN
defaultrouter="192.168.1.254"
ifconfig_em0="dhcp"

# vboxnet3
ifconfig_em1="inet 192.168.103.1 netmask 255.255.255.0"

# routerNet                                                                  
ifconfig_em2="inet 192.168.0.1 netmask 255.255.255.0"

Gateway netstat -rn:
Code:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS         em0
127.0.0.1          link#4             UH          lo0
192.168.0.0/24     link#3             U           em2
192.168.0.1        link#3             UHS         lo0
192.168.1.0/24     link#1             U           em0
192.168.1.194      link#1             UHS         lo0
192.168.103.0/24   link#2             U           em1
192.168.103.1      link#2             UHS         lo0
 
client is able to ping router1, and router1 is able to ping gateway as well as 8.8.8.8 . client can't ping gateway or 8.8.8.8 . My powers of deduction tell me the issue lies somewhere in the configuration of router1. client:~ # traceroute 8.8.8.8 shows the first hop to router1 at 192.168.1.1 and then stalls out. The
Looks like so. What is in /etc/pf.conf?

What is the output of route get 8.8.8.8
 
1. Don't do double NAT as already pointed out. You should preserve the original IP within your local network until it passes the border gateway to the WAN.
2. Don't use the same prefix in different networks (192.168.1.0/24)

It seems your ISP doesn't give you a proper, routable IP address (gateway defaultroute = 192.168.1.254). This will hurt you plenty in the future if you want to use the same rfc1918 prefix internally. Either dump that ISP (my suggestion) and/or use another prefix (e.g. from the 10./8 or 172.16./16 range) for your internal networks to avoid confusion and simplify your routing and firewalling rules. With the current subnetting you won't get a properly working routing setup (multiple NATting isn't a proper solution...) because your gateways/routers have to deal with the same subnet twice, so there is no clear distinction where a packet should be routed.
 
Looks like so. What is in /etc/pf.conf?

What is the output of route get 8.8.8.8
Here's the output:
Code:
   route to: 8.8.8.8
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 192.168.1.1
        fib: 0
  interface: em0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
 
you need to run nat on router1 (and have triple nat !) or change client subnet to other ips
Interesting.

Why can I not use two different 192.168.x.0/24 ranges to distinguish subnets from one another?
 
The gateway machine needs two static routes, or else it won't know how to find 192.168.1.0/24 and 192.168.2.0/24. Router1 needs a static route to find 192.168.2.0/24 and router2 needs a static route to find 192.168.1.0/24.

Gateway:
Code:
static_routes="router1 router2"
route_router1="192.168.1.0/24 192.168.0.2"
route_router2="192.168.2.0/24 192.168.0.3"

Router1:
Code:
static_routes="router2"
route_router2="192.168.2.0/24 192.168.0.3"
defaultrouter="192.168.0.1"

Router2:
Code:
static_routes="router1"
route_router1="192.168.1.0/24 192.168.0.2"
defaultrouter="192.168.0.1"

When looking at routes you always have to look at it from both ways, forward; from the client to the destination address. But you also need to account for the return traffic, from the destination address back to the client. You only set up the routing for the outgoing traffic but the route back isn't set correctly. Packets don't "remember" where they came from, routing is done based solely on the destination address of a packet. And remember that the return traffic will have source and destination addresses reversed.
 
  • Thanks
Reactions: del
2. Don't use the same prefix in different networks (192.168.1.0/24)
The only place I actually need and have configured NAT is on the firewall for gateway. I may have inadvertently enabled NAT, I'm willing to admit that since I am indeed a novice re. the dark arts of networking.

I don't think NAT is the problem here, since I can route to anything in the diagram from client[/gateway] or [icode]server except for gateway.

I did notice that in gateway that the default router for the WAN is using the 192.168.1.0/24 prefix, but since router1 doesn't know that, it shouldn't be an issue, no?
 
I did notice that in gateway that the default router for the WAN is using the 192.168.1.0/24 prefix,
That's the network attached to em0 on the gateway machine? That's a problem. Because that would mean you have two, different, 192.168.1.0/24 networks. That will make routing between them impossible. Make sure each subnet only exists once in your network.

but since router1 doesn't know that, it shouldn't be an issue, no?
How is the gateway machine going to differentiate between the traffic that's on the WAN interface and the traffic that's supposed to go to router1? It can't. So this is not good.
 
  • Thanks
Reactions: del
The gateway machine needs two static routes, or else it won't know how to find 192.168.1.0/24 and 192.168.2.0/24.

Gateway:
Code:
static_routes="router1 router2"
route_router1="192.168.1.0/24 192.168.0.2"
route_router2="192.168.2.0/24 192.168.0.3"


When looking at routes you always have to look at it from both ways, forward; from the client to the destination address. But you also need to account for the return traffic, from the destination address back to the client. You only set up the routing for the outgoing traffic but the route back isn't set correctly. Packets don't "remember" where they came from, routing is done based solely on the destination address of a packet. And remember that the return traffic will have source and destination addresses reversed.
Bloody brilliant! This was an extremely illuminating solve, many thanks. I owe you a pint, Sir!

Summary of the problem and it's solution: I assumed that because the client/server were not pinging successfully, the outbound route was b0rked. I assumed wrongly. The outbound packet was likely reaching its destination just fine, but when traffic needed to return from gateway, gateway didn't know where to forward the traffic for nodes on either the 192.168.1.x or 192.168.2.x subnets. Adding static routes to gateway's /etc/rc.conf for those two subnets fixes the issue.
 
How is the gateway machine going to differentiate between the traffic that's on the WAN interface and the traffic that's supposed to go to router1? It can't. So this is not good.
I see and understand the issue here; my tunnel vision on router1 has caused a number of conflicts. I'll change the subnets to use a different prefix so that gateway can distinguish what nodes belong on which part of the network.
 
I assumed that because the client/server were not pinging successfully, the outbound route was b0rked. I assumed wrongly. The outbound packet was likely reaching its destination just fine, but when traffic needed to return from gateway, gateway didn't know where to forward the traffic for nodes on either the 192.168.1.x or 192.168.2.x subnets.
It's a common and easy mistake to make, only looking at the forward traffic and not routing the return packets correctly. I always imagine myself to be a packet, then look on each "hop" at the routing tables to see what the next hop is, keep going until you reach the endpoint. And from that 'destination' host do the same thing in reverse, looking at the routing table at each hop and figuring out where it should go next. Preferably you'll want the packets to travel through the same hops back to where they came from. On really complex networks you can sometimes end up with an "asynchronous" routing where the return packet takes a different route than the forward packet, try to avoid that as it makes firewalling the traffic on each hop rather complicated (you can't use 'stateful filtering' in that case).
 
Back
Top