Solved PF and/or Routing Question - Possible asymmetric routing situation?

Hello,

I have a FreeBSD 13 system with a single physical interface that I'm using aliases (virtual interfaces?) for additional IP addresses that are in separate subnets. When attempting to send traffic from one alias to another, my traffic dies at SYN_SENT:CLOSED after 4 packets (when viewing from pftop). From Googling, I think this is a asymmetric routing problem as when I disable pf, the traffic flow works and I've verified it's not actually being blocked by any pf rules, so it must be a routing issue right?

I found this guide from pfSense that makes some suggestions about using the state type "sloppy" and I think I'm able to reproduce in pf.conf what they're doing in the GUI (see the bottom of my pf.conf) but the traffic is still failing to traverse between the aliases when pf is enabled. The specific scenario is I'm hosting an IRCD on $ircd_ip and my IRC clients connect from $irc_ip. I could get around this issue by using the same interface for both the IRCD and IRC clients but I like to have them on separate interfaces. Below are copies of all my configuration files:

pftop
Code:
PR        DIR SRC                                           DEST                                                   STATE                AGE       EXP     PKTS    BYTES
tcp       Out 44.55.194.119:41863                          44.55.215.69:6667                              SYN_SENT:CLOSED       00:00:26  00:00:11        4      240


ifconfig
Code:
[root@freebsd ~]# ifconfig
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether fa:16:3e:56:5a:7d
        inet6 fe80::f816:3eff:fe58:5a7c%vtnet0 prefixlen 64 scopeid 0x1
        inet 44.55.245.22 netmask 0xffffff00 broadcast 192.99.245.255
        inet 44.55.215.69 netmask 0xffffffff broadcast 144.217.215.69
        inet 44.55.200.204 netmask 0xffffffff broadcast 142.4.200.204
        inet 44.55.194.119 netmask 0xffffffff broadcast 167.114.194.119
        inet 44.55.177.153 netmask 0xffffffff broadcast 142.44.177.153
        inet 44.55.215.68 netmask 0xffffffff broadcast 144.217.215.68
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 44.55.245.22 netmask 0xffffff00
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog

pf.conf
Bash:
## IRCd Inbound IP Address ##
ircd_ip="44.55.215.69"

## IRC Outbound IP Address ##
irc_ip="44.55.194.119"

## Set and drop these IP ranges on public interface ##
rfc1918 = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
              10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
              0.0.0.0/8, 240.0.0.0/4 }"
 
irc_ports = "{6667, 6697, 7000}"
 
## enable these services ##
int_services = "{domain, ntp, www, https, ssh}"
 
## Skip loop back interface - Skip all PF processing on interface ##
set skip on lo
 
## Sets the interface for which PF should gather statistics such as bytes in/out and packets passed/blocked ##
set loginterface $ext_if

# Deal with attacks based on incorrect handling of packet fragments
scrub in all
 
## Blocking spoofed packets
antispoof quick for $ext_if
 
## Set default policy ##
block return in log all
block out log all
 
## Drop all Non-Routable Addresses ##
block drop in quick on $ext_if from $rfc1918 to any
block drop out quick on $ext_if from any to $rfc1918
 
## Allow Ping-Pong stuff. Be a good sysadmin ##
pass inet proto icmp icmp-type echoreq
 
## Allow essential outgoing traffic ##
pass out quick on $ext_if proto { tcp, udp } to any port $int_services

## Allow IRC Outbound traffic of specific IP ##
pass out quick on $ext_if proto tcp from $irc_ip to any port $irc_ports keep state

## IRCd Allow Rule ##
pass in quick on $ext_if inet proto tcp from any to $ircd_ip port $irc_ports flags any keep state

## Test rules to fix local IRC connections ##
pass quick on $ext_if proto tcp from $ircd_ip to $irc_ip flags any keep state(sloppy)
pass quick on $ext_if from $ircd_ip to $irc_ip keep state(sloppy)

## Test rules to fix local IRC connections ##
pass quick on $ext_if proto tcp from $irc_ip to $ircd_ip flags any keep state(sloppy)
pass quick on $ext_if from $irc_ip to $ircd_ip  keep state(sloppy)

rc.conf
Code:
hostname="freebsd"
ifconfig_DEFAULT="inet 44.55.245.22 netmask 255.255.255.0"
ifconfig_vtnet0_alias0="inet 44.55.215.69 netmask 255.255.255.255"
ifconfig_vtnet0_alias1="inet 44.55.200.204 netmask 255.255.255.255"
ifconfig_vtnet0_alias2="inet 44.55.194.119 netmask 255.255.255.255"
ifconfig_vtnet0_alias3="inet 44.55.177.153 netmask 255.255.255.255"
ifconfig_vtnet0_alias4="inet 44.55.215.68 netmask 255.255.255.255"
growfs_enable="YES"
sshd_enable="YES"
pf_enable="YES"
pf_rules="/usr/local/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
defaultrouter="44.55.245.1"
 
You have two interfaces in 44.55.245.0/24, vtnet0 and lo0. That's a bad idea. Don't put a subnet on more than one interface, that's not going to work. Remove the 44.55.245.22/24 from lo0. Not sure what you're trying to do here but that address is already assigned to vtnet0.

The specific scenario is I'm hosting an IRCD on $ircd_ip and my IRC clients connect from $irc_ip. I could get around this issue by using the same interface for both the IRCD and IRC clients but I like to have them on separate interfaces.
Both addresses are aliases (i.e. additional addresses) on vtnet0, that's one interface.
 
You have two interfaces in 44.55.245.0/24, vtnet0 and lo0. That's a bad idea. Don't put a subnet on more than one interface, that's not going to work.

Do you know why/where this is set? I'm assuming it was done by default because I don't see it in rc.conf. For some reason the "main" IP address 44.55.245.22 is set on both the external interface (vtnet0) and the loopback (lo0) which seems strange to me.
 
Change the ifconfig_DEFAULT to ifconfig_vtnet0. That _DEFAULT is useful but you're assigning aliases to vtnet0 anyway (negating the usefulness of _DEFAULT).
 
Nope, this is basically a fresh install that I modified the pf.conf, rc.conf, added one user and installed unreal ircd & thelounge. I also found this comment on reddit which I think makes sense and explains the problem. The post is about pfsense but I think it's applicable to my situation.

zqpmx
2 points 1 year ago
It has to do with states. Because pfsense is a stateful firewall. ping (ICMP) packets are different than your Oracle connection (TCP) packets. TCP is connection based and creates a formal state in the firewall Each ping packet. Create its own state. But it’s independent. From the previous packet. Stateful firewalls “don’t like” asymmetric routes because paquetes arrive unexpected outside the state and they look like a man in de middle attack. The syn:close is the message. when the connection closes after timeout waiting for a packet it never arrives. Because it took a different route. Pure routers don’t care about asymmetric routes that much, only stateful firewalls ICMP and UDP are not affected. Only TCP.


Code:
[root@freebsd ~]# ps -aux
USER    PID  %CPU %MEM   VSZ   RSS TT  STAT STARTED     TIME COMMAND
root     11 200.0  0.0     0    32  -  RNL  18:36   68:59.56 [idle]
root      0   0.0  0.0     0   496  -  DLs  18:36    0:00.01 [kernel]
root      1   0.0  0.0 11824  1044  -  ILs  18:36    0:00.01 /sbin/init
root      2   0.0  0.0     0    32  -  DL   18:36    0:00.00 [KTLS]
root      3   0.0  0.0     0    16  -  DL   18:36    0:00.00 [crypto]
root      4   0.0  0.0     0    16  -  DL   18:36    0:00.00 [crypto returns 0]
root      5   0.0  0.0     0    16  -  DL   18:36    0:00.00 [crypto returns 1]
root      6   0.0  0.0     0    32  -  DL   18:36    0:00.34 [cam]
root      7   0.0  0.0     0    16  -  DL   18:36    0:00.00 [soaiod1]
root      8   0.0  0.0     0    16  -  DL   18:36    0:00.00 [soaiod2]
root      9   0.0  0.0     0    16  -  DL   18:36    0:00.00 [soaiod3]
root     10   0.0  0.0     0    16  -  DL   18:36    0:00.00 [audit]
root     12   0.0  0.0     0   400  -  WL   18:36    0:01.51 [intr]
root     13   0.0  0.0     0    48  -  DL   18:36    0:00.02 [geom]
root     14   0.0  0.0     0    16  -  DL   18:36    0:00.00 [sequencer 00]
root     15   0.0  0.0     0    80  -  DL   18:36    0:00.03 [usb]
root     16   0.0  0.0     0    16  -  DL   18:36    0:00.00 [soaiod4]
root     17   0.0  0.0     0    16  -  DL   18:36    0:00.54 [rand_harvestq]
root     18   0.0  0.0     0    48  -  DL   18:36    0:00.35 [pagedaemon]
root     19   0.0  0.0     0    16  -  DL   18:36    0:00.00 [vmdaemon]
root     20   0.0  0.0     0    64  -  DL   18:36    0:00.35 [bufdaemon]
root     21   0.0  0.0     0    16  -  DL   18:36    0:00.18 [syncer]
root     22   0.0  0.0     0    16  -  DL   18:36    0:00.02 [vnlru]
root    297   0.0  0.0     0    16  -  DL   18:36    0:00.64 [pf purge]
root    304   0.0  0.1 13532  2992  -  Is   18:36    0:00.00 pflogd: [priv] (pflogd)
_pflogd 308   0.0  0.1 13532  3016  -  S    18:36    0:00.16 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd)
root    435   0.0  0.0 11492  1412  -  Is   18:36    0:00.00 /sbin/devd
root    633   0.0  0.1 12920  2624  -  Is   18:36    0:00.01 /usr/sbin/syslogd -s
root    710   0.0  0.1 12964  2608  -  Is   18:36    0:00.01 /usr/sbin/cron -s
root    733   0.0  0.2 20952  8308  -  Is   18:36    0:00.00 /usr/sbin/sshd
root    743   0.0  0.2 18116  7108  -  Ss   18:36    0:00.04 sendmail: accepting connections (sendmail)
smmsp   746   0.0  0.2 18116  6660  -  Is   18:36    0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
ircd    772   0.0  0.4 40132 17488  -  S    18:36    0:00.56 unrealircd: irc.testnet.local (unrealircd)
root    779   0.0  0.2 21392  9380  -  Ss   18:39    0:00.08 sshd: root@pts/0 (sshd)
root    784   0.0  0.2 21392  9416  -  Ss   18:39    0:00.09 sshd: root@pts/1 (sshd)
root    788   0.0  0.2 21392  9416  -  Ss   18:39    0:00.09 sshd: root@pts/2,pts/3 (sshd)
root    764   0.0  0.1 12892  2268 v0  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv0
root    765   0.0  0.1 12892  2268 v1  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv1
root    766   0.0  0.1 12892  2268 v2  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv2
root    767   0.0  0.1 12892  2268 v3  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv3
root    768   0.0  0.1 12892  2268 v4  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv4
root    769   0.0  0.1 12892  2268 v5  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv5
root    770   0.0  0.1 12892  2268 v6  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv6
root    771   0.0  0.1 12892  2268 v7  Is+  18:36    0:00.00 /usr/libexec/getty Pc ttyv7
root    781   0.0  0.1 14832  4440  0  Ss   18:39    0:00.02 -bash (bash)
root    881   0.0  0.1 13468  2900  0  R+   19:10    0:00.00 ps -aux
root    786   0.0  0.1 14832  4468  1  Is   18:39    0:00.02 -bash (bash)
root    803   0.0  0.1 15976  5168  1  S+   18:40    0:01.52 pftop -f out
root    790   0.0  0.1 14832  4468  2  Ss+  18:39    0:00.02 -bash (bash)
root    808   0.0  0.1 14832  4564  3  Is+  18:42    0:00.01 -bash (bash)
 
Thanks to a kind soul in #freebsd on Libera, they suggested the following fix which worked perfectly.

Code:
scrub in on $ext_if all fragment reassemble
 
Back
Top