PF ruleset for jailed services + NAT

Beeblebrox

Beeblebrox

Finally getting around to expanding my simple pf.conf to greater horizons. I have two NICs (fxp0, alc0). On alc0 (int_if) I have cloned lo1 and aliased two IPs for two separately jailed services (http, mysql). My ifconfig:
Code: 
fxp0: inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
alc0: inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.1.255
lo0:  inet 127.0.0.1 netmask 0xff000000 
lo1:  inet 192.168.2.100 netmask 0xffffff00 
      inet 192.168.2.101 netmask 0xffffff00
Questions:
  1. I want to make jail-http accessible both from int_if and ext_if. Is cloning lo1 the correct way to do it, or have I unnecessarily complicated things?
  2. mysql is needed ONLY for support of jailed http programs and not needed otherwise. Can jail-http communicate directly with jail-sql (with static IP), or is it mandatory to set up the NAT rules in PF? Will ommitting rdr rule for mysql on ext_if break the app running in jail-http for external clients?
  3. There is one server that runs everything I currently need (this is just a lab environment): PF, diskless (dhcp, inetd, NFS, NTP, some others) and jailed http/mysql. No traffic passes from ext_if to int_if with exception for jailed services. Traffic should pass from int --> ext. With these requirements, any corrections to my pf.conf that you might suggest? Also:
    - Do I need anchor ftp?
    - Do I need to rdr from jail-http to ext_if? Should I use TAG instead?
    - Seems I have an error also:
    Code: 
    pfctl: DIOCADDRULE: Address family not supported by protocol family
    /etc/pf.conf:
    Code: 
    int_if="alc0"
ext_if="fxp0"
jhttp="192.168.2.100"
jsql="192.168.2.101"

################ Options ##################################
set block-policy return
set loginterface $ext_if
# set skip on lo # disable

################ Normaliztation ############################
# normalize all incoming traffic. Set ttl 254: limits mapping of hosts behind firewall. Set
random-id to help same. Set mss to ATM network frame size for easy splitting upstream.
scrub on $ext_if all random-id min-ttl 254 max-mss 1452 reassemble tcp fragment reassemble

################ Translation ###############################

# NAT internal hosts
nat on $ext_if from !($ext_if) to any -> ($ext_if:0)
nat on $int_if from lo1:network to any -> ($int_if)

# Redirect internal http/sql traffic to Jail address
rdr on $int_if proto tcp from any to $int_if port http -> $jhttp port http
rdr on $int_if proto tcp from any to $int_if port 3306 -> $jsql port 3306

# Redirect external http/sql traffic to internal Jail address
rdr on $ext_if proto tcp from any to ($ext_if) port http -> $jhttp port http
rdr on $ext_if proto tcp from any to ($ext_if) port 3306 -> $jsql port 3306
no rdr		# DENY rouge redirections

################ Filtering #################################
# loopback
antispoof log quick for lo0 inet
pass quick on lo0 all

# block rules
block in inet6
block in quick on $ext_if from any to 255.255.255.255
block log on $ext_if all
block in from no-route to any
block in from urpf-failed to any
block out log quick on $ext_if from ! $ext_if to any

# pass TCP/UDP, (in+out for ICMP) connections and set state
pass on $ext_if inet proto icmp all icmp-type 8 code 0
pass out quick on $ext_if proto udp from ($ext_if) to any keep state
pass out quick on $ext_if proto tcp from ($ext_if) to any modulate state

# tag packets in on $int_if and pass them out on $ext_if
pass in quick on $int_if from any to any tag INTNET

# Allow http/sql traffic to internal machine ip (alias)
block in on $ext_if
pass in on $ext_if proto tcp from any to $jhttp port http flags S/SA synproxy state
pass in on $ext_if proto tcp from any to $jsql port 3306 flags S/SA synproxy state
pass in quick on $ext_if proto tcp from any to ($ext_if) port ssh  # ssh on host

##### These Seem To Be Not Needed ######
#nat-anchor "ftp-proxy/*"
#pass in quick on $int_if inet proto tcp to any port ftp divert-to 127.0.0.1 port 8021
#nat on $int_if from $jhttp to $ext_if -> ($ext_if)
#match out on egress inet from !(egress:192.168.2.0) to any nat-to (egress:0) # DNW
#pass in on egress inet proto tcp to (egress) port 80 rdr-to $j80 synproxy state
 
I made a quick howto howto setup www in zfs/jail here. Though PF configuration is not explained there, you can get the idea how to set it up.
I personally do create loopback interfaces for jail, seems more logical to me. In my setup all jails can communicate with each other.

Don't forget when TCP communication is used, traffic has to go from int to ext interface, back and forth.
 
Thanks for the Input.
I personally do create loopback interfaces for jail, seems more logical to me.
Click to expand...
Can you please tell me why it's more logical?
Also, am I missing something? You define jail_if="lo666" in pf.conf, but never use it!
 
Beeblebrox said:
Can you please tell me why it's more logical?
Click to expand...

The way I see it: it's a separate network, let it be behind separate interface. It's also safer to do an on-fly modifications, i.e. when you want to add/remove IP address, you do it on virtual interface.
In general, it's easier to maintain. And makes it more clear to write firewall rules too.

The howto was aiming for ZFS/jail setup, PF configuration was there stripped down from my config. It's not used there indeed; NAT is specified with network range there. I use interface name in my real config though.
 
IMO it's an unnecessary complication. Running a jail without 127.0.0.1 won't generate any trouble, as long as jail's IP address for localhost is mapped to the jail's only IP address, defined in $jailroot/etc/hosts. I've tested this successfully with a FAMP server. The ruleset should be easy to implement: block all + pass what you need to access, on $int_if and $ext_if, in and out. Use the logging feature as needed.
 
ecazamir said:
Running a jail without 127.0.0.1 won't generate any trouble,
Click to expand...

But this solution has nothing do to with 127.0.0.1 IP address; I'm creating custom loopback interface, not assigning IP address from 127/8 range. And on the contrary, it's not a complication, it's more like simplification and ease of administration in overall.
 
Code: 
pfctl: DIOCADDRULE: Address family not supported by protocol family
Click to expand...
  • I found the error message source: "block in inet6" was the problem. I have IPv6 disabled system-wide, PF could not match rule to protocol.
  • I have decided to not use lo1_clone and to instead create the aliases (which are also virtual interfaces) directly on alc0, so that Jail IPs resolve to aliased alc0. My current int_if setup (with two aliases): alco: inet 192.168.2.1/24, 192.168.2.100/24, 192.168.2.101/24
  • I will also modify the PF ruleset to allow traffic to jail-sql ONLY from jail-http. Something like this?
    Code: 
    pass in on $ext_if proto tcp from $jhttp to $jsql port 3306  flags S/SA synproxy state
  • At one point I could not connect to the mysql-server running inside the jail. I later realized this is a jail/etc/hosts.allow problem and fixed it. Connects now!
    Code: 
    # mysql -h 192.168.2.101 -u root
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication
packet', system error: 0
# echo "mysqld : 192.168.2.0/255.255.255.0 : allow" > jail/etc/hosts.allow
  • I am getting a lot of blocked outbound traffic:
    Code: 
    # tcpdump -n -e -ttt -r /var/log/pflog
rule 3..16777216/0(match): block out on fxp0: 192.168.1.10.41765 > 74.125.230.243.80: Flags [F.], seq
1133361195, ack 2302451400, win 1041, options [nop,nop,TS val 16562510 ecr 1953727418], length 0
  • I clients on the internal network are able to reach outside so NAT is working for internal clients. However, I need to set the NAMESERVER as gateway's IP. I have placed server/NAT's IP in jail/etc/resolv.conf, added below line to pf.conf, but jailed env still not able to lookup names.
    Code: 
    rdr on $int_if inet proto udp from any to any port 53 -> $ip_gate port 53
  • @matoatlantis: I'm afraid your tutorial does not quite fit my setup. I have two NICs on the machine and the jailed-http service is running on the INTERNAL NIC, and not on the external. In this case PF (NAT) must carry traffic from ext_if to int_if. I have it set up this way because the primary clients for http services are on the internal network, while I'm also granting access to the same services to the external network.
 
@Beeblebrox: Well, yes, your setup is different. I can't help you with MySQL as I don't use SQLs in my setup. I guess this setup of yours is internet facing; my 2¢ to jail topic:

I'm using jails for two purposes: sandbox and security. Sandbox is used just for testing and is like "virtual machine".

I'd treat those SQL/web jails as sort of DMZ and hence create separate interface (loopback) for it with different network. In your case e.g. 192.168.3.0/24. Then, using PF, you can set the rules you need. In other words, I don't like putting jails on the same network as the localnet.

Yes, using PF you can pretty much do the same as with separate loopback. But as I mentioned earlier, I find it easier to maintain, it makes more sense to me and it does ease the administration, specially when root is shared among more administrators :).
 
specially when root is shared among more administrators
Click to expand...
I can see how alias lo1 would make sense in this case - it would be preferable.

I have thought of a different aspect of lo1 also. I think packet switching is done differently when alias lo1 vs alias NIC. It seems to me that the Network Switch must do all the work if you set up alias on NIC, while the packet routing is done in the FreeBSD network stack if alias is on lo1.
More description here.
 
You should take care to network routes and/or ARP problems. Sharing subnets between interfaces may lead to unexpected results.
I agree with @matoatlantis about use of loopback interfaces, but I don't recommend this practice without a strong network experience regarding routing, arp protocol, filtering.
Actually, the kernel uses loopback interface even for local communication with addresses configured on interfaces, at least so says /usr/bin/netstat

Code: 
# ifconfig [color="Red"]bge1[/color]
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 00:xx:xx:9c:xx:xx
	inet [color="Red"]10.16.0.1[/color] netmask 0xfffffc00 broadcast 10.16.3.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

# netstat -nr | less
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
....
10.16.0.0/22       link#2             U           0        0   bge1
[color="Red"]10.16.0.1[/color]          link#2             UHS         0        0    [color="Red"]lo0[/color]
...
# uname -v
FreeBSD 8.2-RELEASE-p2 #3: Sat Jun 11 14:14:14 EEST 2011     root@host.local:/usr/obj/usr/src/sys/CUSTOM
 
I agree with @matoatlantis about use of loopback interfaces, but I don't recommend this practice without a strong network experience regarding routing, arp protocol, filtering.
Click to expand...
Sadly, my knowledge level is pretty weak when it comes to networks.
Sharing subnets between interfaces may lead to unexpected results
Click to expand...
I use same subnet on both NICs, just different address pool. My netstat:
Code: 
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0    14618   fxp0
127.0.0.1          link#3             UH          0        4    lo0
192.168.1.0/24     link#2             U           0      218   fxp0
192.168.1.10       link#2             UHS         0       51    lo0
192.168.2.0/24     link#1             U           0        7   alc0
192.168.2.1        link#1             UHS         0       24    lo0
192.168.2.100      link#1             UHS         0       30    lo0
192.168.2.101      link#1             UHS         0        0    lo0
 
Your subnets are fine, you don't have 192.168.1.0/24 on both alc0 and fxp0.
Actually, FreeBSD does not allow to configure multiple addresses within the same network, unless only one address is specified with the full netmask, the others must be specified with a /32 mask.
Example:
/etc/rc.conf
Code: 
ifconfig_alc0="UP"
ipv4_addrs_alc0="192.168.2.1/24 192.168.2.2/32 192.168.2.3/32"

if you add a supplemental address/32 to another interface, let's say lo1, but the address is within the same subnet, then ARP won't be used for that address on alc0

Wrong example:
Code: 
ifconfig_alc0="UP"
ipv4_addrs_alc0="192.168.2.1/24 192.168.2.2/32 192.168.2.3/32"
ifconfig_re0="UP"
ipv4_addrs_re0="192.168.2.4/32"
In this case, a client connected on alc0 would not be able to reach 192.168.2.4, unless a static route is used on each client.

Wrong example 2:
Code: 
ifconfig_alc0="UP"
ipv4_addrs_alc0="192.168.2.4/24"
ifconfig_re0="UP"
ipv4_addrs_re0="192.168.2.5/24"

The above configuration will fail, because the init scripts will try to add a network route to 192.168.2.0/24 twice. Since FreeBSD is allowing by default a single route per destination, the second /sbin/ifconfig command, executed by the init scripts, will fail.
 
I thought I had solved this but in fact not. While trying to de-bug I had disabled pf. As soon as I re-enabled pf, I got my tftp timeout error on the DC back.

Does not seem to make any difference whether the alias for the jail is set with /24 mask or /32 mask, as both configs work fine when pf is disabled.
 
You must log in or register to reply here.
Back
Top