Multiple NICs and Jail Routing

[darrendavid]

Vesper's two NICs are configured as such:

• igb0:10.0.2.20
• igb1: 10.0.3.20

My plan is to host some normal services on the base OS (mostly things that need read/write access to large fiel trees, media mostly) and configure my jail to host things that will ultimately run through a VPN. So, 10.0.2.20/SERVERLAN services are accessible through inbound port forwarding from the WAN IP, and 10.0.3.21 services do their thing over a VPN. Given that, I want jails to run on igb1/JAILLAN, in particular, one jail at 10.0.3.21, so I can just run all that traffic through the VPN and isolate it from other networks.

Now, I can't seem to isolate the jail traffic to igb1/JAILLAN. When I watch it flowing through pfSense, all of the JAILLAN traffic is running through SERVERLAN. This makes me think that Vesper isn't properly sending the 10.0.3.x packets out the right interface.

I've read about fib as the way to handle multiple interfaces on FreeBSD, and I have the following bits in my rc.conf on Vesper:

# SERVERLAN
ifconfig_igb0="inet 10.0.2.20 netmask 255.255.255.0"
defaultrouter="10.0.2.1"

#JAILLAN
ifconfig_igb1="inet 10.0.3.20 netmask 255.255.255.0 fib 1"
static_routes="jail_if jail_gw"
route_jail_if="-net 10.0.3.0/24 -iface igb1 -fib 1"
route_jail_gw="default 10.0.3.1 -fib 1"

The relevant bits from /etc/jail.conf look like:

webjail {
    host.hostname = "webjail";
    path = "/jailz/webjail";
    exec.fib = "1";
    interface = "igb1";
    ip4.addr += "igb1|10.0.3.21/24";
    allow.raw_sockets;
}

And for purposes of completeness:

> setfib 0 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.2.1           UGS        igb0
10.0.2.0/24        link#1             U          igb0
10.0.2.20          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
[INDENT][/INDENT]
> setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.3.1           UGS        igb1
10.0.3.0/24        link#2             U          igb1
10.0.3.20          link#2             UHS         lo0
10.0.3.21          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               lo0                           UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
ff02::/16                         ::1                           UGRS        lo0

Here's an example from the pf log on pfSense when I try to ping 10.0.3.1 from within webjail:

[BLOCK] Apr 25 11:37:03 SERVERLAN 10.0.3.21 10.0.3.1 ICMP

So you can see traffic that should be on JAILLAN is ending up on SERVERLAN. I'm stumped as to whether my issue is on Vesper or in pfSense (leaning towards the former).

Also, if I run netstat -rn inside the jail, I get an empty response of "Routing tables" with nothing else following. But traffic is indeed heading out, just through the wrong interface.

Thanks in advance for any and all insight here!

Darren

​

 

[darrendavid]

If a packet made it into the logs of pfSense you can be sure that it is blocked there.

Indeed, it's getting blocked there, but that's not the issue. The issue is that the packet from JAILLAN is coming through on SERVERLAN - I'm trying to figure out how to get it to stay on JAILLAN so I can set up the proper rulesets. My hunch is that something's up with my routing tables on Vesper, which is why packets from 10.0.3.0/24 are ending up on 10.0.2.0/24, but I'm not sure how to troubleshoot or fix beyond what I've tried so far.