PF FreeBSD Firewall NAT problems

M

maxchv030796*

Hi guys,

so i have an OVH dedicated server, i installed Proxmox on it. I created 2 VMs, one freebsd VM as a firewall / NAT gateway and one Debian server in the same lan as the firewall with the FreeBSD as Gateway.

On the FreeBSD VM there are 2 net cards, one for the WAN, one for the LAN.

From the FreeBSD i have internet access, i can ping, telnet etc ... all is working from the Firewall / Gateway.

From the Debian server i can ping the internet, i can ping 8.8.8.8.

I have multiples problems :

- From the Debian server, the command telnet 1.1.1.1 53 but telnet 8.8.8.8 53 doesn't works (so i configured /etc/resolv.conf with 1.1.1.1 to be able to use domains but this problem is pretty weird)
- From the Debian server, the command telnet google.com 80 doesn't works (it works from the FreeBSD firewall)
- When i try to connect from my pc with SSH or telnet ip 2223 on the Debian server, i see the packets with tcpdump (on the Debian server) but i have a timeout on the command

Here is what i see on my logs when i do the command doas tcpdump -n -e -ttt -i pflog0 :
Code: 
(telnet 8.8.8.8 53)  00:00:00.000010 rule 19/0(match): pass out on vtnet1: 91.121.40.45.52818 > 8.8.8.8.53: Flags [S], [/S]seq 688104303, win 64240, options [mss 1460,sackOK,TS val 3854927016 ecr 0,nop,wscale 7], length 0
(telnet 1.1.1.1 53)  00:00:00.000010 rule 19/0(match): pass out on vtnet1: 91.121.40.45.53578 > 1.1.1.1.53: Flags , seq 537632035, win 64240, options [mss 1460,sackOK,TS val 3337238453 ecr 0,nop,wscale 7], length 0
(telnet google.com 80)  00:00:00.000004 rule 19/0(match): pass out on vtnet1: 91.121.40.45.60204 > 142.250.178.14.80: Flags , seq 556080316, win 64240, options [mss 1460,sackOK,TS val 2506116294 ecr 0,nop,wscale 7], length 0
(when i connect from the client to the debian server on ssh)  

00:00:01.249859 rule 18/0(match): pass in on vtnet1: my ip.63990 > 192.168.10.10.2223: Flags , seq 2851206258, win 64240, options [mss 1460,sackOK,TS val 3746771 ecr 0,nop,wscale 7], length 0
 00:00:00.000005 rule 17/0(match): pass out on vtnet0: my ip.63990 > 192.168.10.10.2223: Flags , seq 2851206258, win 64240, options [mss 1460,sackOK,TS val 3746771 ecr 0,nop,wscale 7], length 0


Here are my rules :

Code: 
## Macros
ext_if = "vtnet1"     # External interface (WAN)
int_if = "vtnet0"     # Internal interface (LAN)

ext_ip = "91.121.40.45"
int_ip = "192.168.10.1"

# Define the internal server IP
internal_server = "192.168.10.10"
int_network = "192.168.10.0/24"

# Services allowed for outgoing traffic
tcp_pass_out = "{ bootpc, bootps, dhcpv6-client, dhcpv6-server, domain, https, ipp, nicname, ntp, ssh, www, 6667, 6697 }"
udp_pass_out = "{ bootpc, bootps, dhcpv6-client, dhcpv6-server, domain, nicname, ntp }"
icmp_ok_types = "{ echoreq, unreach }"

# Tables for allowed IPs for SSH and FTP access
table <allowed_ssh> persist file "/etc/hh.d/acls/hh_home.txt"
table <allowed_ftp> persist file "/etc/hh.d/acls/hh_home.txt"

table <private> const { 10/8, 172.16/12, 192.168/16 }

## Let free traffic on the loopback interface lo0
set skip on lo0

## Normalization and reassembly of packets
scrub in all

## Translation (NAT for internal network)
nat on $ext_if from $int_network to any -> ($ext_if)
## Port Redirection (for SSH and FTP access)
rdr on $ext_if proto tcp from <allowed_ssh> to $ext_if port 2223 -> $internal_server
rdr on $ext_if proto tcp from <allowed_ftp> to $ext_if port {20, 21, 10000:19999} -> $internal_server

## Filtering

# Default policy: block everything
#block all

### Filters ###
# Permit any packets from internal network to any
pass in on $int_if inet from $int_network to any keep state

## Allow inbound SSH connections from allowed IPs
##Port 2222 firewall freebsd
pass in on $ext_if proto tcp from <allowed_ssh> to $ext_if port 2222 keep state
##Port 2223 web server
pass in on $ext_if proto tcp from <allowed_ssh> to $ext_if port 2223 keep state

## Allow inbound FTP connections (ports 20, 21, passive range 10000-19999) from allowed IPs
pass in on $ext_if proto tcp from <allowed_ftp> to $ext_if port {20, 21, 10000:19999} keep state

## Allow HTTP (80) and HTTPS (443) traffic for everyone
pass in on $ext_if proto tcp from any to $ext_if port { 80 443 } flags S/SA keep state

## Allow DNS and NTP traffic
pass in on $ext_if proto { tcp udp } from any to $ext_if port { domain ntp } keep state

pass out on $int_if from any to $int_if:network keep state

pass out on $ext_if proto tcp all modulate state flags S/SA
pass out on $ext_if proto { udp, icmp } all keep state

# Permit and log all packets from clients in private network through NAT
pass in log on $int_if all
pass out log on $int_if all
pass in log on $ext_if all
pass out log on $ext_if all


Thank you in advance for your advices / help !

Regards,

Maxime
 
Hi,

Thank you for the reply;

I tried scrub in all no-df but i have the same problems when i try to do a telnet 8.8.8.8 53 (telnet 1.1.1.1 53 still works)

for the rdr rule i have this one :

rdr on $ext_if proto tcp from <allowed_ssh> to $ext_if port 2223 -> $internal_server

For the pass rule i have this one :

pass in on $ext_if proto tcp from <allowed_ssh> to $ext_if port 2223 keep state

Do you mean i should change it like that ?

pass in on $ext_if proto tcp from <allowed_ssh> to $internal_server port 2223 keep state

i still have the same problems,

On the debian vm i see the packet from my test but i still have a timeout when i telnet (from the client)
 
Try to disable the hardware offloading on the network interface and verify if promiscuous mode is enabled on the Proxmox
 
Thank you for the reply,

Should i disable hardware offloading and enable promiscuous mode on the proxmox network interface ? Or the FreeBSD network interface ?
 
Disable hardware offloading in the VM (FreeBSD) LRO,TSO rxcsum txcsum etc. and verify if the interface in the proxmox have promiscuous mode enabled.

Side question:
is 10000:19999 range in your setup are for active ftp if yes how did you control the client which random upper port to use and why you don't using ftp proxy?
 
It works ! Thank you so much !

Do you know why i needed to do that ? I mean the technical reason ?

I didn't install the ftp yet, what would you recommand me ? I was planning to use pure-ftpd
 
maxchv030796* said:
Do you know why i needed to do that ? I mean the technical reason ?
Click to expand...

I'm not a driver developer so my understanding may not be correct but here it is:

It depend which hardware offloading is failing. It may be the rx/tx checksum or LRO/TSO (fragmentation and reassembly in pf firewall)

When the hypervisor (HV) reports to the virtual machine (VM) that it support hardware offloading then the virtual network adapter (vNIC) driver enables it.

Let say the rx/tx offloading is enabled on vtnet(4) interface in VM then the checksum on the packet is not calculated by the network driver and it's send as 0x0 / 0xf expecting that the hardware in this case HV will calculate the checksum before sending the frame out but if it fail the entire packet will be discarded on the receiving site. You can test this by sending different combination of IPv4/UDP; TCP; SCTP and IPv6/UDP; TCP packets and observe on the receiving site how are they received.
When hardware offloading is enabled and you are monitoring via tcpdump on the host you will see that all outgoing packets are with invalid tx checksum that's because the driver is not calculating the checksum as it expect this checksum to be put from the hardware before the packet leave the interface it's normal and expected when you are using tcpdump and hw offloading is enabled.

I'm also using pure-ftpd and i'm happy with it but in my case it's standalone ftp server with public ip address so i don't need to forward the active ftp connection using rdr and nat. In your case you need ftp-proxy(8) so it can dynamically open and forward the active ftp connection in your firewall. You can't just forward all upper ports 1024-65535 to your ftp server unless you had a dedicated public ip address for this server and create 1:1 nat or even better use routing.
 
You must log in or register to reply here.
Back
Top