PF Strange (firewall?) issue

megapearl · Nov 27, 2018

Hi,

I have 2 sites, both running FreeBSD 11.2-RELEASE-p4, they are connected via IPSec site2site pfsense VM appliances.
I can't connect to port 443 from Site A (10.0.1.x range) to Site B (10.0.0.x range), also the other way around I can't connect from Site B to Site A.
All other ports between both sites are no problem, both are running SSH, Apache and a lot of other services.
Tried to disable all firewalls, disabled packet filtering in both pfSense VM's.
Reinstalled Apache on both sites.

I can however connect from a host within Site B to apache on the FreeBSD server at Site B.
I can also connect from a host within Site A to apache on FreeBSD server at Site A.

From Site A when I run links https://10.0.0.2/network I got a timeout.
From Site A when I run links http://10.0.0.2/network it got connected and serving the webpage.

Packet is arriving at site B, but why do I get a timeout?

When I use tcpdump at site B and make a connection to port 443 with links http and https at site A:

Code:

root@fileserver:/etc # tcpdump -n "src host 10.0.1.2 and dst host 10.0.0.2 and (dst port 80 or dst port 443)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:47:02.890222 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [S], seq 1554715555, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269739209 ecr 0], length 0
20:47:02.904127 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [.], ack 2970669685, win 129, options [nop,nop,TS val 269739228 ecr 559453973], length 0
20:47:02.906458 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [P.], seq 0:597, ack 1, win 129, options [nop,nop,TS val 269739228 ecr 559453973], length 597: HTTP: GET /network HTTP/1.1
20:47:02.925144 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [S], seq 4136953711, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269739248 ecr 0], length 0
20:47:02.942442 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [.], ack 3326231012, win 129, options [nop,nop,TS val 269739259 ecr 3163961707], length 0
20:47:02.942565 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [P.], seq 0:617, ack 1, win 129, options [nop,nop,TS val 269739259 ecr 3163961707], length 617: HTTP: GET /network/ HTTP/1.1
20:47:03.021286 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [.], ack 516, win 129, options [nop,nop,TS val 269739340 ecr 559453984], length 0
20:47:03.060824 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [.], ack 1107, win 129, options [nop,nop,TS val 269739381 ecr 3163961728], length 0
20:47:07.963072 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [.], ack 517, win 129, options [nop,nop,TS val 269744280 ecr 559459023], length 0
20:47:07.986907 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [.], ack 1108, win 129, options [nop,nop,TS val 269744309 ecr 3163966758], length 0
20:47:22.985644 IP 10.0.1.2.27036 > 10.0.0.2.80: Flags [F.], seq 617, ack 1108, win 129, options [nop,nop,TS val 269759298 ecr 3163966758], length 0
20:47:22.985695 IP 10.0.1.2.58865 > 10.0.0.2.80: Flags [F.], seq 597, ack 517, win 129, options [nop,nop,TS val 269759298 ecr 559459023], length 0

Port 80 is working

Code:

root@fileserver:/etc # tcpdump -n "src host 10.0.1.2 and dst host 10.0.0.2 and (dst port 80 or dst port 443)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:47:57.356404 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269793678 ecr 0], length 0
20:48:00.358021 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269796679 ecr 0], length 0
20:48:03.579332 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269799903 ecr 0], length 0
20:48:06.775710 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269803098 ecr 0], length 0
20:48:09.977563 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269806300 ecr 0], length 0
20:48:13.181119 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269809504 ecr 0], length 0
20:48:19.382248 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269815699 ecr 0], length 0
20:48:31.581173 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269827901 ecr 0], length 0
20:48:55.781616 IP 10.0.1.2.56363 > 10.0.0.2.443: Flags [S], seq 3731676877, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269852099 ecr 0], length 0
20:49:12.362492 IP 10.0.1.2.14159 > 10.0.0.2.443: Flags [S], seq 2556223907, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269868684 ecr 0], length 0
20:49:15.365536 IP 10.0.1.2.14159 > 10.0.0.2.443: Flags [S], seq 2556223907, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269871689 ecr 0], length 0
20:49:18.581915 IP 10.0.1.2.14159 > 10.0.0.2.443: Flags [S], seq 2556223907, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 269874900 ecr 0], length 0

Port 443 gives a timeout

Apache does not log anything.
No dropped or rejected packets at both pfSense firewalls.

SSH and all other port from Site A to Site B are connecting without any problem.

When I temporary change the 443 port in apache config to 444 at site B it connects fine from site A to that port.

Anyone have a clue?

Best Regards,
Donald.

ShelLuser · Nov 27, 2018

When you say "cannot connect" you basically mean that it gives a timeout, right?

What does sockstat -4l | grep 443 tell you?

Did you actually set up SSL at all or just told Apache to listen on 443 (just asking to rule out possible causes)?

megapearl · Nov 27, 2018

Yes, SSL is working fine and setup correctly, apache (ssl) is reachable on port 443 from any other host within lan and from the outside via NAT.
It is only the FreeBSD Virtual Machine which isn't reachable and only on port 443 from vpn clients via pfSense. port 80 and 22 for example are connecting fine.
The Apache daemon on Ubuntu Server which is on the same network is connecting fine on port 443 via that same vpn.

Code:

root@fileserver:/home/donald # sockstat -4l | grep 443
www      httpd      43775 6  tcp4   10.0.0.2:443          *:*
www      httpd      43750 6  tcp4   10.0.0.2:443          *:*
www      httpd      43707 6  tcp4   10.0.0.2:443          *:*
www      httpd      43659 6  tcp4   10.0.0.2:443          *:*
www      httpd      43558 6  tcp4   10.0.0.2:443          *:*
www      httpd      40974 6  tcp4   10.0.0.2:443          *:*
www      httpd      33369 6  tcp4   10.0.0.2:443          *:*
www      httpd      2584  6  tcp4   10.0.0.2:443          *:*
www      httpd      2577  6  tcp4   10.0.0.2:443          *:*
www      httpd      2554  6  tcp4   10.0.0.2:443          *:*
www      httpd      2552  6  tcp4   10.0.0.2:443          *:*
root     httpd      2490  6  tcp4   10.0.0.2:443          *:*

ShelLuser · Nov 28, 2018

So what's in /etc/pf.conf? You apparently already suspected a firewall but you didn't provide any further details. If the package arrives at the server yet Apache isn't responding to it then a filter somewhere seems like a logical conclusion.

megapearl · Nov 28, 2018

Code:

int_if       = "vmx0"
ipv4_net     = "10.0.0.0/24"
ipv6_net     = "2001:470:7f85::/64"
tcp_services = "{ssh, smtp, smtps, domain, rtip, sunrpc, http, https, lockd, netbios-ssn, microsoft-ds, afpovertcp, nfsd, sip, sip-tls, 554, 957, ipp, 2812, 3306, 5038, 6881, 6789, 6791, 7878, 8081, 8181, 8182, 8333, 8443, 8888, 8889, 8989, 9091, 9117, 27017}"
udp_services = "{domain, ntp, sip, radius, rtip, sunrpc, nfsd, rmc, lockd, tftp, mdns, 6881, netbios-ns, netbios-dgm, syslog, 5100:5200, 5353}"
tcp_state    = "flags S/SAFR modulate state"
udp_state    = "keep state"
set debug urgent
set require-order yes
set block-policy return
set loginterface $int_if
set state-policy if-bound
set fingerprints "/etc/pf.os"
set ruleset-optimization none
set skip on lo0
set optimization normal
#set timeout { tcp.established 3600, tcp.closing 60 }
#set limit { states 20000, frags 20000, src-nodes 2000 }
#scrub log on $int_if all random-id set-tos 0x1c fragment reassemble max-mss 1440 min-ttl 64

anchor "f2b/*"
block in  log on $int_if
block out log on $int_if

pass in  on $int_if inet  proto icmp  from any       to any
pass in  on $int_if inet  proto tcp   from any       to any port $tcp_services $tcp_state
pass in  on $int_if inet  proto udp   from any       to any port $udp_services $udp_state

pass in  on $int_if inet6 proto icmp6 from any       to any $udp_state
pass in  on $int_if inet6 proto tcp   from any       to any
pass in  on $int_if inet6 proto udp   from any       to any $udp_state

pass out on $int_if inet  proto icmp  from any to any $udp_state
pass out on $int_if inet  proto tcp   from any to any $tcp_state
pass out on $int_if inet  proto udp   from any to any $udp_state

pass out on $int_if inet6 proto icmp6 from any to any $udp_state
pass out on $int_if inet6 proto tcp   from any to any
pass out on $int_if inet6 proto udp   from any to any $udp_state

But I also turned pf off using service pf stop but it doesn't make any difference.
Fail2ban is also turned off and 10.0.1.x and 10.0.0.x ranges are excluded.

jiml8 · Nov 30, 2018

Take a look at what rules are actually being written. Pfsense will write a set of default rules and it could well be that one of those is biting you.

pfctl -sa

and see what it says.

kpa · Dec 1, 2018

No, PF doesn't have any default rules other than a hidden "pass all" rule that sets the default policy if no rules are loaded.

sko · Dec 1, 2018

megapearl said:
apache (ssl) is reachable on port 443 from any other host within lan and from the outside via NAT.

Can you show us that NAT rule? Are you using different gateways for NAT and the IPsec tunnel?
If you only NAT 443, I suspect you are also NATing the traffic through your IPsec tunnel, which is breaking the return path. Don't tcpdump with source/destination filters but with "host" filters so you can see inbound and outbound traffic and spot routing/NAT problems.

megapearl · Mar 29, 2019

Still struggling with this issue,

Other hosts within the same lan on both ends are no problem with any ports, only the FreeBSD VM's are, for example I have setup Apache on a Ubuntu VM (10.0.0.x range) with port 443 open, and I can connect fine from the FreeBSD VM on the other side (10.0.1.x range).
Tried to remove all custom stuff in /etc/pfctl.conf and /boot/loader.conf and booted with default options, and booted with GENERIC kernel instead of the custom one on both FreeBSD servers, this make no difference.
I don't see any dropped or rejected packages in both pfSense VM's or FreeBSD VM's.

I had to attach the output of the pfctl -sa commands as files to post it here. (message too large error from FreeBSD forum)

pfctl -sa on FreeBSD at 10.0.0.2 = PFsense fileserver.txt
pfctl -sa on FreeBSD at 10.0.1.2 = PFsense mainserver.txt
pfctl -sa on pfSense at 10.0.0.1 = Pf-fileserver-freebsd.txt
pfctl -sa on pfSense at 10.0.1.1 = Pf-mainserver-freebsd.txt

A lot of code, hope someone can help me out.

Best Regards,
Donald.

SirDice · Mar 29, 2019

Check your IPSec policies. These can silently drop connections.

megapearl · Mar 29, 2019

I can't find anything wrong with the IPSec policies, they are accepting the net2net subnets to each other.

When I use tcpdump on FreeBSD 10.0.0.2 and on FreeBSD 10.0.1.2 while opening links https://10.0.1.2/ from FreeBSD 10.0.0.2, I see:

FreeBSD 10.0.0.2 (while initiating connection to 10.0.1.2 port 443 with links)

Code:

root@fileserver:/ # tcpdump -ni vmx0 host 10.0.1.2 and dst port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:08:21.582791 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456513468 ecr 0], length 0
16:08:24.587133 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456516480 ecr 0], length 0
16:08:27.786755 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456519680 ecr 0], length 0
16:08:31.000127 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456522891 ecr 0], length 0
16:08:34.206617 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456526090 ecr 0], length 0
16:08:37.415703 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456529304 ecr 0], length 0
16:08:43.617127 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456535509 ecr 0], length 0
16:08:55.851518 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456547730 ecr 0], length 0
16:09:20.046542 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 456571930 ecr 0], length 0

FreeBSD 10.0.1.2

Code:

root@mainserver:/ # tcpdump -ni vmx0 host 10.0.0.2 and dst port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:08:21.626688 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456513468 ecr 0], length 0
16:08:24.627307 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456516480 ecr 0], length 0
16:08:27.827717 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456519680 ecr 0], length 0
16:08:31.040831 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456522891 ecr 0], length 0
16:08:34.247298 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456526090 ecr 0], length 0
16:08:37.456412 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456529304 ecr 0], length 0
16:08:43.658043 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456535509 ecr 0], length 0
16:08:55.891695 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456547730 ecr 0], length 0
16:09:20.087060 IP 10.0.0.2.14695 > 10.0.1.2.443: Flags [S], seq 2416575032, win 65535, options [mss 1428,nop,wscale 9,sackOK,TS val 456571930 ecr 0], length 0

And the other way around, connecting with links to 10.0.0.2 from 10.0.1.2

Code:

root@fileserver:/ # tcpdump -ni vmx0 host 10.0.1.2 and dst port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:14:48.674496 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4751884 ecr 0], length 0
16:14:51.686884 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4754900 ecr 0], length 0
16:14:54.894030 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4758104 ecr 0], length 0
16:14:58.092061 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4761305 ecr 0], length 0
16:15:01.290460 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4764504 ecr 0], length 0
16:15:04.494679 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4767709 ecr 0], length 0
16:15:10.691560 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4773905 ecr 0], length 0
16:15:22.897219 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4786110 ecr 0], length 0
16:15:47.096819 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4810305 ecr 0], length 0

Code:

root@mainserver:/ # tcpdump -ni vmx0 host 10.0.0.2 and dst port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:14:48.697892 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4751884 ecr 0], length 0
16:14:51.709844 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4754900 ecr 0], length 0
16:14:54.912990 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4758104 ecr 0], length 0
16:14:58.114925 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4761305 ecr 0], length 0
16:15:01.313707 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4764504 ecr 0], length 0
16:15:04.518111 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4767709 ecr 0], length 0
16:15:10.714690 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4773905 ecr 0], length 0
16:15:22.919230 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4786110 ecr 0], length 0
16:15:47.119200 IP 10.0.1.2.44177 > 10.0.0.2.443: Flags [S], seq 3936678378, win 65535, options [mss 1416,nop,wscale 9,sackOK,TS val 4810305 ecr 0], length 0

Can you see anything odd here?

PF Strange (firewall?) issue

megapearl

ShelLuser

megapearl

ShelLuser

megapearl

jiml8

kpa

sko

megapearl

Attachments

SirDice

Administrator

megapearl