Using tagged VLANs within jails

Good day.

I have had no issues with routing in jails; it all works fine.

I have had no issues with tagged VLANs; works as expected.

I cannot however figure out how to get tagged VLANs working within jails.

My configuration is as such:

Code:
ifconfig_em0="inet 10.0.253.7 netmask 255.255.255.0"
ifconfig_em1="up"
cloned_interfaces="vlan0 vlan1 bridge0"
ifconfig_vlan0="inet 10.0.240.7 netmask 255.255.255.0 vlan 240 vlandev em1"
ifconfig_vlan1="inet 10.0.255.7 netmask 255.255.255.0 vlan 255 vlandev em1"
ifconfig_bridge0="addm vlan1 up"
defaultrouter="10.0.253.1"

The output is as such:

Code:
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 00:25:90:dc:ca:47
        inet 10.0.253.7 netmask 0xffffff00 broadcast 10.0.253.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 00:25:90:dc:ca:46
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4000403<RXCSUM,TXCSUM,LRO,NOMAP>
        ether 00:25:90:dc:ca:46
        inet 10.0.240.7 netmask 0xffffff00 broadcast 10.0.240.255
        groups: vlan
        vlan: 240 vlanproto: 802.1q vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4000403<RXCSUM,TXCSUM,LRO,NOMAP>
        ether 00:25:90:dc:ca:46
        inet 10.0.255.7 netmask 0xffffff00 broadcast 10.0.255.255
        groups: vlan
        vlan: 255 vlanproto: 802.1q vlanpcp: 0 parent interface: em1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:c8:13
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

From the jail, I can ping itself, and the default gateway. I cannot ping further. The default gateway is a Juniper SRX running NAT, and it's been running fine with basically the same configuration for the past decade (I've upgraded devices, but pretty much dropped code from the older device into the newer device, as needed).

When listing jails, I see the correct IP listed:

Code:
[root@server01 /home/thatchrisp]# jls
   JID  IP Address      Hostname                      Path
     1  10.0.255.7      plex                          /usr/jails/plex

No issues pinging itself:

Code:
# ping 10.0.255.7
PING 10.0.255.7 (10.0.255.7): 56 data bytes
64 bytes from 10.0.255.7: icmp_seq=0 ttl=64 time=0.020 ms
64 bytes from 10.0.255.7: icmp_seq=1 ttl=64 time=0.022 ms
64 bytes from 10.0.255.7: icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from 10.0.255.7: icmp_seq=3 ttl=64 time=0.020 ms
^C
--- 10.0.255.7 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.020/0.022/0.027/0.003 ms
#

No issues pinging the host's default gateway (I realize that this is in the same broadcast domain, so it doesn't tell us much, other than the rest of the broadcast domain can at least see it, and confirms the switch port configuration is correct):

Code:
# ping 10.0.255.1
PING 10.0.255.1 (10.0.255.1): 56 data bytes
64 bytes from 10.0.255.1: icmp_seq=0 ttl=64 time=0.542 ms
64 bytes from 10.0.255.1: icmp_seq=1 ttl=64 time=0.579 ms
64 bytes from 10.0.255.1: icmp_seq=2 ttl=64 time=0.547 ms
64 bytes from 10.0.255.1: icmp_seq=3 ttl=64 time=0.508 ms
^C
--- 10.0.255.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.508/0.544/0.579/0.025 ms

I cannot however ping the host's default gateway:

Code:
# ping 10.0.253.1
PING 10.0.253.1 (10.0.253.1): 56 data bytes
^C
--- 10.0.253.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

I feel like the issue lies here, so I setup the bridge, per a few blogs I read, but that hasn't fixed the issue.

Here's a bit more information:

Code:
FreeBSD server01 13.2-RELEASE-p3 FreeBSD 13.2-RELEASE-p3 GENERIC amd64

Switch port config for good measure:

Code:
show interfaces ge-0/0/8   
description "SERVER01: IGB1";
gratuitous-arp-reply;
ether-options {
    auto-negotiation;
    link-mode full-duplex;
    speed {
        1g;
    }
}
unit 0 {
    family ethernet-switching {
        port-mode trunk;
        vlan {
            members [ WIRELESS UNIFI ];
        }
    }
}

Wireless is VLAN 255, and Unifi is VLAN 240.

Anyone have any suggestions?

Thanks in advance!
 
In my opinion you do not need a bridge. Bridge with single interface has no sence.
Can you show jail's configuration file?
Do you use vnet for jails?
Can you provide wiring&network diagram with all relevant info?

As far as I know, jails use preconfigured IPs on the on the host and jails not relevant to VLAN's.
It should be no difference for jails betwen physical interface and vlan interface.

Try to ping everything from host, but specify additional ping options like -S
ping()
Code:
     -S src_addr
             Use the following IP address as the source address in outgoing
             packets.  On hosts with more than one IP address, this option can
             be used to force the source address to be something other than
             the IP address of the interface the probe packet is sent on.  If
             the IP address is not one of this machine's interface addresses,
             an error is returned and nothing is sent.
 
I cannot however ping the host's default gateway:
Packets are sent to the router at 10.0.255.1, it routes the packets further to 10.0.253.1. This issue is on your router, not the FreeBSD host.
 
Packets are sent to the router at 10.0.255.1, it routes the packets further to 10.0.253.1. This issue is on your router, not the FreeBSD host.
It's not. 10.0.255.1 and 10.0.253.1 are the same Juniper SRX. There is no reason for traffic to forward from 10.0.255.1 to 10.0.253.1; they are one in the same in the RIB and FIB. Traffic within each of these broadcast domains works as expected; this is not a greenfield network.

For giggles I added a security policy going from VLAN 255 to VLAN 253, allowing all traffic, and it did not change the behaviour.

Code:
firewall# run show security policies from-zone WIRELESS to-zone MEDIA
From zone: WIRELESS, To zone: MEDIA
  Policy: WIRELESS-TO-MEDIA, State: enabled, Index: 14, Scope Policy: 0, Sequence number: 1
    Source addresses: any
    Destination addresses: any
    Applications: any
    Action: permit

firewall# run show security policies from-zone WIRELESS to-zone WAN     
From zone: WIRELESS, To zone: WAN
  Policy: WIRELESS-TO-WAN, State: enabled, Index: 5, Scope Policy: 0, Sequence number: 1
    Source addresses: any
    Destination addresses: any
    Applications: any
    Action: permit

Adding this rule did not change anything unfortunately.

Traffic work on the server works as expected, firewall included, until I bind a VLANed IP to a jail. If I remove the IP from the VLAN and assign it to the jail it works fine, so long as the switch port is untagged, and the .1q statement is removed.
 
It's not. 10.0.255.1 and 10.0.253.1 are the same Juniper SRX.
Check the routing engine there. And the policies, because you're moving from one subnet to another.

There is no reason for traffic to forward from 10.0.255.1 to 10.0.253.1;
The jail only has a gateway to 10.0.255.1, so that's where the packets end up getting sent to. It's the router's job at 10.0.255.1 to route the traffic further towards 10.0.253.1.
 
Check the routing engine there. And the policies, because you're moving from one subnet to another.

As pasted above, there's a valid, entirely open policy from one zone to another.

The jail only has a gateway to 10.0.255.1, so that's where the packets end up getting sent to. It's the router's job at 10.0.255.1 to route the traffic further towards 10.0.253.1.

Also as mentioned above, 10.0.255.1 and 10.0.253.1 are the same device; it cannot forward it further.

Code:
firewall# run show route table inet.0 | match 10.0.253.1 
10.0.253.1/32      *[Local/0] 3d 12:03:41

firewall# run show route table inet.0 | match 10.0.255.1   
10.0.255.1/32      *[Local/0] 3d 12:03:44

Furthermore, the firewall can ping all of the IPs bound to the server, so it knows how to get back to the source.

Code:
firewall# run ping 10.0.240.7 
PING 10.0.240.7 (10.0.240.7): 56 data bytes
64 bytes from 10.0.240.7: icmp_seq=0 ttl=64 time=7.540 ms
64 bytes from 10.0.240.7: icmp_seq=1 ttl=64 time=2.474 ms
64 bytes from 10.0.240.7: icmp_seq=2 ttl=64 time=2.845 ms
^C
--- 10.0.240.7 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.474/4.286/7.540/2.306 ms

firewall# run ping 10.0.253.7   
PING 10.0.253.7 (10.0.253.7): 56 data bytes
64 bytes from 10.0.253.7: icmp_seq=0 ttl=64 time=5.978 ms
64 bytes from 10.0.253.7: icmp_seq=1 ttl=64 time=2.774 ms
64 bytes from 10.0.253.7: icmp_seq=2 ttl=64 time=2.510 ms
^C
--- 10.0.253.7 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.510/3.754/5.978/1.576 ms

firewall# run ping 10.0.255.7   
PING 10.0.255.7 (10.0.255.7): 56 data bytes
64 bytes from 10.0.255.7: icmp_seq=0 ttl=64 time=7.862 ms
64 bytes from 10.0.255.7: icmp_seq=1 ttl=64 time=2.395 ms
64 bytes from 10.0.255.7: icmp_seq=2 ttl=64 time=2.583 ms
^C
--- 10.0.255.7 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.395/4.280/7.862/2.534 ms
 
Also as mentioned above, 10.0.255.1 and 10.0.253.1 are the same device; it cannot forward it further.
That is irrelevant from the jail's perspective. It can only reach 10.0.255.1. That's the device's problem to solve. Routing only looks at the destination address, and only sends it to the 'next' hop. It has no notion of what happens beyond that. It's the receiving router's job to process it further, if needed.
 
In my opinion you do not need a bridge. Bridge with single interface has no sence.
Can you show jail's configuration file?
Do you use vnet for jails?
Can you provide wiring&network diagram with all relevant info?

I've removed the bridge, re-added it with more interfaces, etc, no difference.

The jail config is very vanilla.

Code:
export jail_plex_hostname="plex"
export jail_plex_ip="10.0.255.7"
export jail_plex_rootdir="/usr/jails/plex"
export jail_plex_exec_start="/bin/sh /etc/rc"
export jail_plex_exec_stop=""
export jail_plex_mount_enable="YES"
export jail_plex_devfs_enable="YES"
export jail_plex_devfs_ruleset="devfsrules_jail"
export jail_plex_procfs_enable="YES"
export jail_plex_fdescfs_enable="YES"
export jail_plex_image=""
export jail_plex_imagetype=""
export jail_plex_attachparams=""
export jail_plex_attachblocking=""
export jail_plex_forceblocking=""
export jail_plex_zfs_datasets=""
export jail_plex_cpuset=""
export jail_plex_fib=""
export jail_plex_parentzfs=""
export jail_plex_parameters=""
export jail_plex_post_start_script=""
export jail_plex_retention_policy=""
export jail_plex_parameters="allow.raw_sockets=1"

I am not using VNET.

As far as I know, jails use preconfigured IPs on the on the host and jails not relevant to VLAN's.
It should be no difference for jails betwen physical interface and vlan interface.

Try to ping everything from host, but specify additional ping options like -S
ping()
Code:
     -S src_addr
             Use the following IP address as the source address in outgoing
             packets.  On hosts with more than one IP address, this option can
             be used to force the source address to be something other than
             the IP address of the interface the probe packet is sent on.  If
             the IP address is not one of this machine's interface addresses,
             an error is returned and nothing is sent.

I don't have a diagram handy, but the traffic flows as follows:

Juniper SRX 340 -> Juniper EX3400 VC cluster -> FreeBSD server

The VC cluster is purely switching, all the routing is done on the SRX.

There are three interfaces on the server
  1. BMC (IPMI) which is connected to an entirely different VLAN and has no bearing on the networking issue
  2. em0, which is an untagged port, with the host IP, 10.0.253.7
  3. em1, which is a tagged port with the two VLANs (240, 10.0.240.7 & 255, 10.0.255.7) on it
There are no issues with em0/10.0.253.7. I can ping 8.8.8.8 when sourcing with it.

Code:
# ping -S 10.0.253.7 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.0.253.7: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=59 time=43.297 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=26.621 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=44.293 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=42.026 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 26.621/39.059/44.293/7.226 ms

As you can see, the firewall can see the host and the VLANs:
Code:
firewall# run show arp | match 10.0.253.7  
00:25:90:dc:ca:47 10.0.253.7      10.0.253.7                vlan.253            none

firewall# run show arp | match 10.0.240.7  
00:25:90:dc:ca:46 10.0.240.7      10.0.240.7                vlan.240            none

firewall# run show arp | match 10.0.255.7  
00:25:90:dc:ca:46 10.0.255.7      10.0.255.7                vlan.255            none

When trying to ping anything outside the broadcast domain when using the -S flag, it fails. Pinging anything inside that broadcast domain works.

Code:
# ping -S 10.0.255.7 10.0.255.1
PING 10.0.255.1 (10.0.255.1) from 10.0.255.7: 56 data bytes
64 bytes from 10.0.255.1: icmp_seq=0 ttl=64 time=0.529 ms
64 bytes from 10.0.255.1: icmp_seq=1 ttl=64 time=0.538 ms
^C
--- 10.0.255.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.529/0.534/0.538/0.005 ms


# ping -S 10.0.255.7 10.0.240.1
PING 10.0.240.1 (10.0.240.1) from 10.0.255.7: 56 data bytes
^C
--- 10.0.240.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


# ping -S 10.0.240.7 10.0.240.1
PING 10.0.240.1 (10.0.240.1) from 10.0.240.7: 56 data bytes
64 bytes from 10.0.240.1: icmp_seq=0 ttl=64 time=0.549 ms
64 bytes from 10.0.240.1: icmp_seq=1 ttl=64 time=0.508 ms
^C
--- 10.0.240.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.508/0.529/0.549/0.020 ms


# ping -S 10.0.240.7 10.0.255.1
PING 10.0.255.1 (10.0.255.1) from 10.0.240.7: 56 data bytes
^C
--- 10.0.255.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

The exception to this is pinging the other VLAN host, which it should be able to, since it's in the FIB:

Code:
ping -S 10.0.240.7 10.0.255.7
PING 10.0.255.7 (10.0.255.7) from 10.0.240.7: 56 data bytes
64 bytes from 10.0.255.7: icmp_seq=0 ttl=64 time=0.019 ms
64 bytes from 10.0.255.7: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 10.0.255.7: icmp_seq=2 ttl=64 time=0.021 ms
^C
--- 10.0.255.7 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.019/0.022/0.026/0.003 ms


ping -S 10.0.255.7 10.0.240.7
PING 10.0.240.7 (10.0.240.7) from 10.0.255.7: 56 data bytes
64 bytes from 10.0.240.7: icmp_seq=0 ttl=64 time=0.014 ms
64 bytes from 10.0.240.7: icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from 10.0.240.7: icmp_seq=2 ttl=64 time=0.018 ms
^C
--- 10.0.240.7 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.014/0.017/0.020/0.002 ms

FIB:

Code:
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.253.1         UGS         em0
10.0.240.0/24      link#4             U         vlan0
10.0.240.7         link#4             UHS         lo0
10.0.253.0/24      link#1             U           em0
10.0.253.7         link#1             UHS         lo0
10.0.255.0/24      link#5             U         vlan1
10.0.255.7         link#5             UHS         lo0
127.0.0.1          link#3             UH          lo0

Normally in a situation like this, one would assume that the source route isn't installed on the next hop, but it definitely is, as per above. The hop after that is the ISP, and everything is NAT'ed so that won't be the issue.

I have devices currently connected to all three VLANs that are working. There is working NAT, and security policies on all zones/VLANs. Also as mentioned, this is a mature network with very few incidents over the past many years, mostly because I am using enterprise gear.

As everything checks out on the next hop, the issue appears be within the FreeBSD FIB. It appears it simply doesn't know how to forward traffic from that broadcast domain to the next hop. It's behaving as if there is no default route for the VLANed subnets. I've read many posts on the forums and other sites now that suggest that jails, by default, use the host FIB. It seems to be mangling the source, or not permitting traffic back to the source from outside the broadcast domain.
 
That is irrelevant from the jail's perspective. It can only reach 10.0.255.1. That's the device's problem to solve. Routing only looks at the destination address, and only sends it to the 'next' hop. It has no notion of what happens beyond that. It's the receiving router's job to process it further, if needed.

I'm sure you understand FreeBSD, but I'm not sure you understand routing. I'll walk you through this.

Source IP is 10.0.255.7. Next hop, the firewall, is 10.0.253.1 and 10.0.255.1. The are in separate broadcast domains, but as long as there are security policies permitting the traffic between them, and there are, then that's all that is needed. In short, it cannot forward a transient packet from itself, to itself. It just needs a valid next hop at that point, which will be the WAN interface with NAT on it, so the source IP is now that the of the WAN interface of the firewall. The ISPs router obviously knows the how to get back to my firewall, which is in the same broadcast domain, and their FIB as a directly connected route.

To illustrate my point:

Code:
3c:61:04:d6:6d:48 10.0.253.1      10.0.253.1                vlan.253                none
3c:61:04:d6:6d:48 10.0.255.1      10.0.255.1                vlan.255                none

There is no routing to 10.0.253.1 required; it is one and the same, with valid routes installed, and the firewall sees everything as intended, as per my previous reply to im.
 
but I'm not sure you understand routing.
Having done network management since the early 90's, I'm fairly sure I know how this works.

Source IP is 10.0.255.7. Next hop, the firewall, is 10.0.253.1
That's it. So packets get sent to 10.0.253.1 and it is assumed the router at 10.0.253.1 knows how to transport it to 10.0.255.1. How it does this or where it gets moved to is completely invisible from the jail's (10.0.255.7) perspective.

The are in separate broadcast domains, but as long as there are security policies permitting the traffic between them, and there are, then that's all that is needed.
I assume each broadcast domain, i.e. each subnet is on separate security zones. The policies simply tell it what is or isn't allowed to move from one zone to another. The Juniper's routing engine determines where to transfer those packets to.

What does show configuration routing-instances | display set tell you on the Juniper?

it cannot forward a transient packet from itself, to itself.
That's why it's generally a bad idea to put two (or more) different subnets on the same interface. While technically possible, you can and will run into issues like this.

There is no routing to 10.0.253.1 required;
Of course routing is required. The only difference here is that 10.0.253.0/24 and 10.0.255.0/24 are directly connected networks and their routes are implied.
 
Having done network management since the early 90's, I'm fairly sure I know how this works.

While I admire and respect your FreeBSD knowledge, I believe that you may have a low level understanding of networking.

That's it. So packets get sent to 10.0.253.1 and it is assumed the router at 10.0.253.1 knows how to transport it to 10.0.255.1. How it does this or where it gets moved to is completely invisible from the jail's (10.0.255.7) perspective.

Incorrect. There is no transporting it. It's a transient packet. It cannot forward it from itself to itself. I've already stated this. The destination IP is already in the FIB, and directly connected.

I assume each broadcast domain, i.e. each subnet is on separate security zones. The policies simply tell it what is or isn't allowed to move from one zone to another. The Juniper's routing engine determines where to transfer those packets to.

Yes, all subnets are all on separate zones, for security reasons. Jails aren't enough. I've already demonstrated that the Juniper is not the issue. It knows how to find everything, as per my previous reply.

What does show configuration routing-instances | display set tell you on the Juniper?

Nothing. I'm not running routing instances. I have no need for l3vpn, separate RIBs, or any other valid reason for using routing instances on the firewall.

That's why it's generally a bad idea to put two (or more) different subnets on the same interface. While technically possible, you can and will run into issues like this.

This happens more often than not. It's not a bad idea in general at all. In fact, it's how my firewall is configured.

Of course routing is required. The only difference here is that 10.0.253.0/24 and 10.0.255.0/24 are directly connected networks and their routes are implied.

You're not hearing me. Routing is not needed between 10.0.253.1 (not the subnet, the IP) and 10.0.255.1 (again, not the subnet, the IP) because they both belong to the same device.

After some more troubleshooting, I've found the issue.

As a default route or routing policies per jail has not been created (looking into vnet now), the packets are sent out the wrong zone:

Code:
Session ID: 11332, Policy name: MEDIA-TO-WAN/6, Timeout: 2, Valid
  In: 10.0.255.7/4 --> 8.8.8.8/60681;icmp, If: vlan.253, Pkts: 1, Bytes: 84
  Out: 8.8.8.8/60681 --> 159.196.223.213/1391;icmp, If: fe-0/0/6.0, Pkts: 0, Bytes: 0

Temporarily, I have setup NAT and everything is working, but this feels like a poor solution. I think the proper answer is with vnet, or some other mechanism to change the routing table within the jail and send the traffic out the proper interface.

Code:
Session ID: 28032, Policy name: MEDIA-TO-WAN/6, Timeout: 4, Valid
  In: 10.0.253.7/4 --> 8.8.8.8/55301;icmp, If: vlan.253, Pkts: 1, Bytes: 84
  Out: 8.8.8.8/55301 --> 159.196.223.213/22735;icmp, If: fe-0/0/6.0, Pkts: 1, Bytes: 84
 
Back
Top