PF Stuck in SYN_SENT

Daniel Santos · Jan 20, 2016

Hello all,

I am currently working in setting up a new network for third parties to reach some of our internal systems through dedicated lines. The idea is to use OpenBGPd+PF+CARP to have a fully redundant solution. My current configuration is explained in the attached files.

The problem is that when I try to connect to some server in the 10.100.0.0/24 network, PF never goes from SYN_SENT to ESTABLISHED, although tpcdump(8) shows that all handshake packets have passed.
For example, when I try a ssh connection to 10.100.0.1 here is what tcpdump and pfctl tells me.

Code:

sudo pfctl -ss | grep -A 3 ":22"
...
[B]all tcp 10.100.0.1:22 <- 10.120.200.18:50301       CLOSED:SYN_SENT
all tcp 10.120.200.18:50301 -> 10.100.0.1:22       SYN_SENT:CLOSED[/B]
...

inbound interface tcpdump:

Code:

tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:00.000000 00:22:19:7a:99:57 > 28:80:23:ac:7c:f4, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 126, id 7886, offset 0, flags [DF], proto TCP (6), length 52)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [[B]S[/B]], cksum 0xa101 (correct), seq 445641177, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:00.002445 00:22:19:7a:99:57 > 28:80:23:ac:7c:f4, ethertype IPv4 (0x0800), length 56: (tos 0x0, ttl 126, id 7887, offset 0, flags [DF], proto TCP (6), length 40)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [[B].[/B]], cksum 0x723d (correct), ack 464351962, win 256, length 0
00:00:00.014019 00:22:19:7a:99:57 > 28:80:23:ac:7c:f4, ethertype IPv4 (0x0800), length 82: (tos 0x0, ttl 126, id 7888, offset 0, flags [DF], proto TCP (6), length 68)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [P.], cksum 0x7d0b (correct), seq 0:28, ack 1, win 256, length 28
...

outbound interface tcpdump:

Code:

tcpdump: listening on vlan1620, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:00.000000 28:80:23:ac:7c:f4 > 00:0c:29:bd:86:05, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 125, id 7886, offset 0, flags [DF], proto TCP (6), length 52)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [[B]S[/B]], cksum 0xa101 (correct), seq 445641177, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:00.002421 28:80:23:ac:7c:f4 > 00:0c:29:bd:86:05, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 125, id 7887, offset 0, flags [DF], proto TCP (6), length 40)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [[B].[/B]], cksum 0x723d (correct), ack 464351962, win 256, length 0
00:00:00.013995 28:80:23:ac:7c:f4 > 00:0c:29:bd:86:05, ethertype IPv4 (0x0800), length 82: (tos 0x0, ttl 125, id 7888, offset 0, flags [DF], proto TCP (6), length 68)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [P.], cksum 0x7d0b (correct), seq 0:28, ack 1, win 256, length 28
00:00:00.033068 28:80:23:ac:7c:f4 > 00:0c:29:bd:86:05, ethertype IPv4 (0x0800), length 726: (tos 0x0, ttl 125, id 7889, offset 0, flags [DF], proto TCP (6), length 712)
    10.120.200.18.50336 > 10.100.0.1.22: Flags [P.], cksum 0x373d (correct), seq 28:700, ack 22, win 256, length 672

And to have a better prove that I have a TCP circuit in place, the ssh client gives me a working prompt, at least while the 30 seconds that PF holds the SYN_SENT connection in the state table.

The strangest thing, is that connections coming from inside the 10.100.0.0/24 network works as expected and if I move to static routes everything works properly.

Might be a kernel or PF bug?

SirDice · Jan 20, 2016

Please post your pf.conf.

Daniel Santos · Jan 20, 2016

SirDice

Here is my pf.conf.

Code:

#INTERFACES AND GROUPS
if_accessNetwork="vlan2"
if_internet="vlan1130"
if_remote_peer_ISP1="vlan1620"
if_remote_peer_ISP2="vlan1621"
ifg_remote_peer = "alo_serv"
ifg_bgp = "bgp_interfaces"

#OPTIONS
set skip on lo0
set limit states 256000
set limit frags 1024000
set limit table-entries 1024000
set require-order yes
set optimization normal
set state-policy floating
set debug urgent

#OBJECTS
hst_local_fw_dc_vlan2_vip="10.10.10.100"
hst_local_fw01_dc_vlan2_vip="10.10.10.101"
hst_local_fw02_dc_vlan2_vip="10.10.10.102"
hst_local_fw_dc_vip="201.7.143.34"
hst_local_fw01_dc_ext="201.7.143.35"
hst_local_fw02_dc_ext="201.7.143.36"
hst_proc_wpad="10.151.2.51"
hst_proc_cclocalvirtual="10.211.156.7"
hst_proc_cclocal06="10.211.156.9"
hst_eptc_srv6="10.160.1.76"
hst_localAd1="10.150.150.151"
hst_localAd2="10.150.150.152"
hst_localAd3="10.150.150.157"
hst_admin1_workstation="10.120.200.10"
hst_admin2_workstation="10.120.200.18"
hst_broadcast="255.255.255.255"
hst_local_pkg_server="10.210.10.11"
hst_internet_router="10.40.44.1"
hst_remote_peer_ad01="10.100.0.10"

grp_localDomainControllers="{" \
$hst_localAd1 \
$hst_localAd2 \
$hst_localAd3 \
"}"

grp_segInfoWorkstations="{" \
$hst_admin1_workstation \
$hst_admin2_workstation \
"}"

grp_pkgRepoServers="{" \
  $hst_local_pkg_server \
"}"

grp_ntopClients="{" \
  $hst_internet_router \
"}"

grp_remote_peer_DomainControllers="{" \
  $hst_remote_peer_ad01 \
"}"

grp_datacenterFirewalls="{" \
  $hst_local_fw_dc_vlan2_vip \
  $hst_local_fw01_dc_vlan2_vip \
  $hst_local_fw02_dc_vlan2_vip \
  $hst_local_fw_dc_vip \
  $hst_local_fw01_dc_ext \
  $hst_local_fw02_dc_ext \
"}"

table <tbl_bruteForceHosts> persist #Dynamic table used to track and block brute-force hosts

pool_default_nat_pool="{" \
  $hst_local_fw_dc_vip \
"}"

net_localAcesso="10.10.10.0/24"
net_localProducao="10.120.0.0/16"
net_localProxies="201.7.143.64/28"
net_localNetAdmin="10.120.200.0/24"
net_localContactCenter="10.211.156.0/24"
net_remote_peer_Servers="10.100.0.0/25"
net_remote_peer_Workstations="10.100.0.128/25"
net_rfc990_loopback="127.0.0.0/8"
net_rfc1700_0="0.0.0.0/8"
net_rfc1918_192="192.168.0.0/16"
net_rfc1918_172="172.16.0.0/12"
net_rfc1918_10="10.0.0.0/8"
net_rfc2544_192="198.18.0.0/15"
net_rfc3068_192="192.88.99.0/24"
net_rfc3927_apipa="169.254.0.0/16"
net_rfc5736_192="192.0.0.0/24"
net_rfc5737_192_0="192.0.2.0/24"
net_rfc5737_192_51="198.51.100.0/24"
net_rfc5737_203="203.0.113.0/24"
net_ipv4_multicast="224.0.0.0/4"
net_rfc6598_100="100.64.0.0/10"
net_rfc6890_240="240.0.0.0/4"

table <tbl_rfc1918> persist { \
$net_rfc1918_192 \
$net_rfc1918_10 \
$net_rfc1918_172 \
}

table <tbl_ipv4ReservedAddresses> persist { \
$net_rfc1918_192 \
$net_rfc1918_10 \
$net_rfc1918_172 \
$net_rfc990_loopback \
$net_rfc1700_0 \
$net_rfc2544_192 \
$net_rfc3068_192 \
$net_rfc3927_apipa \
$net_rfc5736_192 \
$net_rfc5737_192_0 \
$net_rfc5737_192_51 \
$net_rfc5737_203 \
$net_ipv4_multicast \
$net_rfc6598_100 \
$net_rfc6890_240 \
}

table <tbl_remote_peer_nets> persist{ \
  $net_remote_peer_Servers \
  $net_remote_peer_Workstations \
\ }

#Services
svc_safe_icmp_types="{echoreq,echorep}"
svc_unsafe_icmp_types="{unreach}"
svc_webBrowsing="{http,https,ftp}"
svc_netbios="{netbios-ns,netbios-dgm,netbios-ssn}"
svc_bootp="{bootps,bootpc}"
svc_dhcpv6="{dhcpv6-client,dhcpv6-server}"
svc_dhcp_all="{bootps,bootpc,dhcpv6-client,dhcpv6-server}"
svc_email = "{smtp,pop3,imap,imap3,imaps,pop3s}"
svc_activeDirectoryTCP = "{ldap,ldaps,domain,kerberos-sec,ms-ad-gc,135,microsoft-ds}"
svc_activeDirectoryUDP = "{ldap,domain,kerberos-sec,microsoft-ds}"
svc_thema_db = "{1521}"
svc_proc_voip_rtp = "{ 10000:20000 }"
svc_proc_softPhoneAuth = "{ 8080 }"
svc_winFileShare = "{netbios-ssn, microsoft-ds}"

#Source tracking options
sto_default = "keep state (max 90000,source-track rule,max-src-conn 1000,max-src-nodes 256)"
sto_ssh = "keep state (max 20,source-track rule,max-src-conn 10,max-src-nodes 100,max-src-conn-rate 100/30, overload <tbl_bruteForceHosts> flush global)"
sto_ssh_fw = "keep state (max 20,source-track rule,max-src-conn 10,max-src-nodes 100,max-src-conn-rate 100/30,overload <tbl_bruteForceHosts> flush global,no-sync)"
sto_web = "keep state (max 4096,source-track rule,max-src-conn 64,max-src-nodes 512,max-src-conn-rate 500/100,overload <tbl_bruteForceHosts> flush global)"
sto_nosync = "keep state (no-sync)"


#NAT
no nat from <tbl_ipv4ReservedAddresses> to <tbl_ipv4ReservedAddresses>
nat pass on $if_internet inet proto {tcp,udp,icmp} from $net_rfc1918_192 to !<tbl_ipv4ReservedAddresses> -> $pool_default_nat_pool round-robin sticky-address

#Antispoof
antispoof for $if_internet
antispoof for $if_accessNetwork

#Firewall filtering Rules
pass quick proto carp all keep state (no-sync) label "Allow CARP"
pass log quick inet proto icmp from <tbl_rfc1918> to <tbl_rfc1918> icmp-type $svc_safe_icmp_types label "Allow all ICMP echo requests: rfc1918 <-> rfc1918"
pass log quick inet proto icmp from <tbl_rfc1918> to !<tbl_ipv4ReservedAddresses> icmp-type $svc_safe_icmp_types label "Allow all ICMP echo requests: rfc1918 -> Internet"
block drop in quick on vlan1130 from any to $hst_proc__vip label "Silently drop packets to  VIP"
block in log quick on $if_internet from <tbl_ipv4ReservedAddresses> to any label "Block incoming outside packets with invalid addresses"
block out log quick on $if_internet from any to <tbl_ipv4ReservedAddresses> label "Block outgoing packets with invalid addresses"
block in log from <tbl_bruteForceHosts> to any label "Early drop the brutals"
pass in log quick on $ifg_bgp inet proto tcp from any to (self) port bgp label "Allow incoming BGP connections"
pass out log quick on $ifg_bgp inet proto tcp from (self) to any port bgp label "Allow outgoing BGP connections"
pass in log quick on $if_internet inet proto esp from !<tbl_ipv4ReservedAddresses> to ($if_internet) label "Allow ESP IN"
pass out log quick on $if_internet inet proto esp from ($if_internet) to !<tbl_ipv4ReservedAddresses> label "Allow ESP OUT"
pass in log quick on $if_internet inet proto udp from !<tbl_ipv4ReservedAddresses> to ($if_internet) port {500,4500} label "Allow NAT-T IN"
pass out log quick on $if_internet inet proto udp from ($if_internet) to !<tbl_ipv4ReservedAddresses> port {500,4500} label "Allow NAT-T OUT"
pass in log quick inet proto tcp from $grp_segInfoWorkstations to ($if_accessNetwork) port = 3000 $sto_nosync label "NTOP Web interface"
pass in log quick inet proto udp from $grp_ntopClients to ($if_accessNetwork) port = 2055 $sto_nosync label "NTOP collector"
pass in log quick inet proto tcp from $grp_segInfoWorkstations to { ($if_accessNetwork),($if_internet)} port ssh $sto_ssh_fw label "Allow SSH access"
pass log quick inet proto udp from any to { ($if_accessNetwork), ($if_servers)} port = 56789 label "PFTABLED"
pass log quick inet proto udp from (self) to $grp_datacenterFirewalls port = 56789 label "PFTABLED"
pass out log quick inet proto tcp from self to <tbl_rfc1918> port = ssh $sto_nosync label "Allow ssh access to all private addresses"
pass out log quick inet proto icmp from self to any icmp-type $svc_safe_icmp_types $sto_nosync label "Allow all ICMP echo requests from the firewall itself"
pass out log quick on $if_internet inet proto tcp from ($if_internet) to !<tbl_ipv4ReservedAddresses> port = 43 $sto_nosync label "Allow WHOIS queries"
pass out log quick inet proto udp from ($if_accessNetwork) to $grp_localDomainControllers port = ntp $sto_nosync label "Allow NTP queries"
pass out log quick inet proto udp from ($if_accessNetwork) to $grp_localDomainControllers port = domain $sto_nosync label "Allow DNS queries"
pass out log quick inet proto tcp from ($if_accessNetwork) to $net_localProxies port = squid $sto_nosync label "Allow web access through proxies"
pass out log quick on $if_internet inet proto tcp from ($if_internet) to !<tbl_ipv4ReservedAddresses> port $svc_webBrowsing $sto_nosync label "Allow direct web browsing"
pass out log quick on $if_accessNetwork inet proto tcp from ($if_accessNetwork) to $grp_pkgRepoServers port {http https} $sto_nosync label "Allow package updating and installation"
pass out log quick on $if_accessNetwork inet proto tcp from ($if_accessNetwork) to $hst_local_pkg_server port {ssh} $sto_nosync label "Update configuration through git"
pass out log quick on $if_accessNetwork inet proto tcp from ($if_accessNetwork) to $hst_local_pkg_server port {http, https} $sto_nosync label "Update packages"
pass out log quick on $if_internet inet proto udp from ($if_internet) to !<tbl_ipv4ReservedAddresses> port = domain $sto_nosync label "Allow DNS queries"
pass in log quick on $if_accessNetwork inet proto tcp from $grp_segInfoWorkstations to $hst_local_fw_dc_vlan2_vip port {ssh,http,https} label "Allow management connections to proc-vm-lab"
block return quick log from any to self label "Stealth"

#Remote peer network filtering
anchor "alo_inbound" inet from any to <tbl_remote_peer_nets> {
  pass in log quick on $if_accessNetwork inet proto tcp from $grp_segInfoWorkstations to <tbl_remote_peer_nets> port {ssh rdp http https} label "Allow Management Connections"
  pass out log quick on $ifg_remote_peer inet proto tcp from $grp_segInfoWorkstations to <tbl_remote_peer_nets> port {ssh rdp http https} label "Allow Management Connections"
  pass log quick inet proto tcp from $grp_segInfoWorkstations to <tbl_remote_peer_nets> port {netbios-ssn microsoft-ds} label "File transfers"
  anchor "servers" inet from any to $net_remote_peer_Servers {
     anchor "active_directory" inet from any to $grp_remote_peer_DomainControllers {
      pass log quick inet proto tcp from $grp_localDomainControllers to $grp_remote_peer_DomainControllers port $svc_activeDirectoryTCP label "Active Directory TCP Services"
      pass log quick inet proto udp from $grp_localDomainControllers to $grp_remote_peer_DomainControllers port $svc_activeDirectoryUDP label "Active Directory UDP Services"
      pass log quick inet proto tcp from $grp_localDomainControllers to $grp_remote_peer_DomainControllers port >= 1023 label "RPC TCP high ports"
     }
   }

  anchor "workstations" inet from any to $net_remote_peer_Workstations {
     anchor "voip" inet from $net_localContactCenter to $net_remote_peer_Workstations {
      pass log quick inet proto udp from $hst_proc_cclocalvirtual to $net_remote_peer_Workstations port sip label "SIP session stablishment with Asterisk"
      pass log quick inet proto tcp from $hst_proc_cclocalvirtual to $net_remote_peer_Workstations port $svc_proc_voip_rtp label "RTP communication with Asterisk"
    }
  }
}
anchor "alo_inbound/*"
anchor "alo_inbound/servers/*"
anchor "alo_inbound/workstations/*"

anchor "alo_outbound" inet from <tbl_remote_peer_nets> to any {
  anchor "servers" inet from $net_remote_peer_Servers to any {
    anchor "active_directory" inet from $grp_remote_peer_DomainControllers to any {
      pass log quick inet proto tcp from $grp_remote_peer_DomainControllers to $grp_localDomainControllers port $svc_activeDirectoryTCP label "Active Directory TCP Services"
      pass log quick inet proto udp from $grp_remote_peer_DomainControllers to $grp_localDomainControllers port $svc_activeDirectoryUDP label "Active Directory UDP Services"
    }
    anchor "web_proxy" inet from $net_remote_peer_Servers to $net_localProxies {
      pass log quick inet proto tcp from $net_remote_peer_Servers to $net_localProxies port squid label "Web browsing through proxies"
      pass log quick inet proto tcp from $net_remote_peer_Servers to $hst_proc_wpad port http label "wpad download"
    }
  }

  anchor "workstations" inet from $net_remote_peer_Workstations to any {
    anchor "web_proxy" inet from $net_remote_peer_Workstations to $net_localProxies {
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $net_localProxies port squid label "Web browsing through proxies"
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $hst_proc_wpad port http label "wpad download"
    }
    anchor "thema" inet from $net_remote_peer_Workstations to $hst_eptc_srv6 {
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $hst_eptc_srv6 port $svc_winFileShare label "Access to bde_central and etpc_dk file shares"
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $hst_eptc_srv6 port $svc_thema_db label "Access to system database"
    }
    anchor "voip" inet from $net_remote_peer_Workstations to $net_localContactCenter {
      pass log quick inet proto udp from $net_remote_peer_Workstations to $hst_proc_cclocalvirtual port sip label "SIP session stablishment with Asterisk"
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $hst_proc_cclocalvirtual port $svc_proc_voip_rtp label "RTP communication with Asterisk"
      pass log quick inet proto tcp from $net_remote_peer_Workstations to $hst_proc_cclocal06 port $svc_proc_softPhoneAuth label "Softphone authentication"
    }
  }
}

anchor "alo_outbound/*"
anchor "alo_outbound/servers/*"
anchor "alo_outbound/workstations/*"


block drop quick proto {tcp,udp} from any to any port $svc_netbios label "Silently drop unwanted NetBIOS packets"
block drop quick proto {tcp,udp} from any to any port $svc_dhcp_all label "Silently drop unwanted DHCP packets"
block drop quick from any to $net_ipv4_multicast label "Silently drop unwanted multicast packets"
block return log label "Cleanup"

Thanks for your help!

Daniel Santos · Jan 20, 2016

SirDice

Please consider this thread closed. It was just a silly assymetric routing issue that I overlooked.

Thanks again for the interest.

PF Stuck in SYN_SENT

Daniel Santos

Attachments

SirDice

Administrator

Daniel Santos

Daniel Santos