PF Jails with NAT

Shadow53

New Member

Reaction score: 2
Messages: 13

I'm trying to set up an Ampache media server at home and am taking the opportunity to learn how to use jails on FreeBSD. I'm trying to set up jails on a separate loopback network on the host and use the NAT features of PF to direct the traffic where it should go. I've tried following multiple different guides for this, except everything I can find refers to using ezjail and I want to do it using just jail.conf and pf.conf, if possible.

The problem is that the jails cannot access the internet. I have set up their resolv.conf files and - I thought - set up the shared network and the NAT rules to make it work. Here's my current setup:

/etc/rc.conf
Code:
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="faustus"
ifconfig_fxp0="DHCP"
ifconfig_fxp0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpdate_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
kld_list="/boot/kernel/i915kms.ko"
cloned_interfaces="lo1"
#ifconfig_lo1="inet 10.0.1.1/24 netmask 255.255.255.0"
ipv4_addrs_lo1="10.0.1.1/24 10.0.1.2/24 10.0.1.3/24"
# Enabled packet forwarding between interfaces
gateway_enable="YES"
blacklistd_enable="YES"
jail_enable="YES"
pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
/etc/jail.conf
Code:
mount.devfs;
path="/usr/jails/$name";
host.hostname="$name.localdomain";
exec.clean;
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";
exec.consolelog="/usr/jails/$name/var/log/jail.log";
interface="lo1";

nginx {
        ip4.addr="lo1|10.0.1.2";
}

ampache {
        ip4.addr="lo1|10.0.1.3";
}
/etc/pf.conf
Code:
ext_if="fxp0"
ext_net=$ext_if:network
jail_if="lo1"
jail_net=$jail_if:network

nginx_ports="{ http, https }"
nginx_ip="10.0.1.2"

# Don't filter loopback
set skip on lo0
set loginterface lo1
set block-policy return
set fail-policy  return

# Sanitize incoming data
scrub in on $ext_if all

# Route HTTP/S to nginx jail
#nat pass on $ext_if inet from $jail_net to any -> ($ext_if)
nat on $ext_if from $jail_net to any -> ($ext_if)
rdr on $ext_if proto tcp from any to ($ext_if) port $nginx_ports -> $nginx_ip

# Allow blacklistd to block stuff
#anchor "blacklistd/*" in on $ext_if
# Block incoming by default
#block in
# Allow outgoing by default
pass out
pass in
# Prevent spoofing attacks
#antispoof for $ext_if

# Allow traffic to/from jails
#pass in on $ext_if proto tcp from $ext_net to $jail_net port $nginx_ports keep state
#pass out on $jail_if proto tcp from $ext_net to $jail_net port $nginx_ports keep state
# Allow SSH
#pass in on $ext_if proto tcp from any to ($ext_if) port ssh
# Allow ICMP
#pass inet proto icmp from $jail_net to any keep state
pass in on $ext_if inet proto icmp to ($ext_if) icmp-type { unreach, redir, timex, echoreq }
pass in on $jail_if inet proto icmp to ($jail_if) icmp-type { unreach, redir, timex, echoreq }
Running tcpdump on pflog0 shows nothing when I attempt to use the host's pkg to install packages in a jail. The host has internet access. The jails' rc.conf files only disable sendmail and prevents syslogd from communicating on the network. The jails properly receive their IP address and hang when they attempt to access the internet, followed eventually by "No address record" for requests with domain names or "Operation timed out" for requests using IP addresses.

I'm new to FreeBSD and PF, so I'm not sure where to go from here. It seems like PF is blocking part of the request, response, or both, but I'm not sure how to check that or how I would fix it. Any help is appreciated.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

Run tcpdump(1) on the $ext_if interface, then initiate some outgoing traffic from one of the jails. Check if there's actually anything going out and if the source address is correctly NAT'ed. Using tcpdump(1) allows you to look at the actual packets, invaluable tool when dealing with firewalls or connection problems.
 

gkontos

Daemon

Reaction score: 475
Messages: 2,147

If you want to monitor the pflog0 then you need to use[ FILE] log[/FILE] in your statements :


Code:
pass out log
pass in log
 

sko

Well-Known Member

Reaction score: 221
Messages: 430

I'm usually using the default loopback interface (lo0) for jails on hosts with only one IP, but besides that my NAT/rdr rules look the same, so I'm not fully convinced it is a filter/rdr/NAT problem.


Usually you don't pre-set the jails IP in the rc.conf of the host, at least that's what I've done on several jailhosts with NAT and one IP.
I only set one IP on that interface via the rc.conf that isn't connected to any jail. This IP is used by the host to reach the associated subnet on that interface. This IP is also used as the default GW by the jails.
The jail IPs are added to the interface when the jails are started.

Make sure your jails have a default route (=the hosts IP on the same loopback IF; e.g. 10.0.1.1) and resolver configured and reachable.

Then try to ping to/from the host and/or a separate host in the same network. If you still don't get any traffic from/to the jails, tcpdump on $ext_if as well as lo0 (and on the outside host you are pinging to/from), so you can find out where packets are either blocked or not properly routed and they are lost.
If ICMP gets the full way in one direction but no answers are generated, also look at ARP traffic and if requests/replies are transmitted the full way in both directions.
 
OP
OP
S

Shadow53

New Member

Reaction score: 2
Messages: 13

Again, I'm new to this sort of thing, so I couldn't really tell how to read the tcpdump output. It looks like the jail is successfully sending requests over $ext_if but isn't receiving any response. It queries the first DNS server for a domain, then the second server, then the first again for a different domain... I was attempting to install a package in the jail, so it was querying multiple pkg servers.

I was able to ping the host from the jail and the jail from the host. I was also able to ping 1.1.1.1 from the host, though attempting to ping a domain caused the same problem as the above paragraph.

Looks like the DNS server is responding to the host, but the host isn't properly directing the response to the jail. I tried a naive attempt to fix this by adding the following line to pf.conf:

Code:
rdr on $ext_if proto { tcp, udp } from any to ($ext_if) port domain -> ($jail_if)
That didn't work. I'm pretty sure that's not the correct way to redirect DNS response packets, but I thought I'd try. I feel like I shouldn't need to redirect response packets though, since PF is a stateful firewall. If I'm understanding how that works, I shouldn't need to do anything about the response packets, right?
 

gkontos

Daemon

Reaction score: 475
Messages: 2,147

Try replacing this:

nat on $ext_if from $jail_net to any -> ($ext_if) to this:
nat on $ext_if from 10.0.1.2 to any -> ($ext_if)

Assuming that 10.0.1.2 is the IP address of the jail
 
OP
OP
S

Shadow53

New Member

Reaction score: 2
Messages: 13

I decided to try spinning up a Digital Ocean droplet with FreeBSD and set up jails there. It looks like being on a home network with a NAT-ing router is causing some issues, as I have functional jail networking on the droplet but not on the home server. Both are using nearly identical configurations, with the only difference being that I can't hardcode the ip address for the home server.

I think I have an ebook copy of the Book of PF somewhere, so I might try reading through that and see if I'm missing something...
 
Top