ezjail networking

I usually enjoy learning by doing, however, this is somehow a production server (people need it running). I’m in hurry to help my friend and I already managed to lock out myself once. Please help.

The goal is to have a jail for an untrusted user. The host is directly connected to the internet. The host has a single public IP address (nothing like 192.168.x.x), which I will replace to 199.199.199.199 for security reasons.

The jail should:
* See the internet
* Run Apache
* Be reachable via ssh on the host’s IP (on other port than 22)
* See the host’s MySQL server (TCP/IP will be fine)
* Be safe

The host also runs Apache so I need reverse proxy setup instead of port forwarding. The host uses ipfw. I’m not asking how to configure Apache or MySQL, the sole issue is the networking part.

At the beginning I found two links:

http://www.bsdnow.tv/tutorials/jails
https://www.freebsd.org/doc/handbook/jails-ezjail.html

I went with the latter as I felt it more secure, even though I have little idea about aliasing Ethernet interfaces.

ezjail-admin create oi 'lo1|127.0.1.1,em0|192.168.1.244'
Code:
# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:30:48:f6:45:d0
  inet 199.199.199.199 netmask 0xffffff80 broadcast 199.199.199.198
  inet 192.168.1.244 netmask 0xffffffff broadcast 192.168.1.244
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (100baseTX <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet 127.0.1.1 netmask 0xffffffff
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Host’s /etc/rc.conf:
Code:
ifconfig_em0="inet 199.199.199.199 netmask 255.255.255.128 broadcast 199.199.199.198"
defaultrouter="199.199.199.197"

# Firewall
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

# Ezjail
ezjail_enable="YES"
cloned_interfaces="${cloned_interfaces} lo1"
/usr/local/etc/ipfw.rules:
Code:
# set command
IPF="ipfw -q add"

# clear
ipfw -q -f flush

#loopback
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any

# open port ftp (20,21), ssh (22), mail (25)
# http (80), dns (53) etc
$IPF 110 allow tcp from any to any 21 in
$IPF 120 allow tcp from any to any 22 in
$IPF 130 allow tcp from any to any 25 in
$IPF 140 allow udp from any to any 53 in
$IPF 150 allow tcp from any to any 53 in
$IPF 160 allow tcp from any to any 80 in
$IPF 170 allow tcp from any to any 143 in
$IPF 180 allow tcp from any to any 465 in
$IPF 190 allow tcp from any to any 993 in


# allow any out
$IPF 400 allow all from any to any out

# deny and log everything
$IPF 500 deny log all from any to any
The jail’s /etc/hosts:
Code:
#::1 localhost oi.my.domain oi
127.0.1.1 localhost oi.my.domain oi
I’ve commented out the ipv6 part since I had no better idea.

At this point the jail doesn’t see the internet. I’m aware that ping shouldn’t work and it’s nothing to do with the resolv.conf since I try to ssh <another-server-reachable-from-the-host-and-anywhere>, and it waits forever inside the jail. Same about installing pkg or telnet <whatever> 80.

There is nothing in the jail’s rc.conf yet.

I would like to understand whether I need NAT at this point. I guess I’ll need it anyway to reach the jail from outside via ssh, however, do I need it also to see the internet from the jail?

I read this too: https://forums.freebsd.org/threads/howto-quick-setup-of-jail-on-zfs-using-ezjail-with-pf-nat.30063/

I haven’t tried as I’m not a network expert and I would like to understand what I am doing instead of copying dozens of lines into the config files when I don’t necessarily need them. I would like to keep it as simple as possible.

However, I tried:
/etc/rc.d/ipfw stop
on the host. It didn’t help.

Finally I added this to the rc.conf on the host:
Code:
gateway_enable="YES"
natd_enable="YES"
natd_interface="em0"
natd_flags="-dynamic -m"
firewall_nat_enable="YES"

No luck. By the way I see nothing in the firewall log regarding to the outgoing ssh attempts.

Please tell:
* Whether I really need NAT for outgoing connections from the jail.
* How to make a minimal config to forward the host’s 1022 port to the jail’s 22 port via ipfw.

I also think that the host’s and the jail’s sshd are fighting for the 192.168.1.244 IP’s ssh port with the default config (bound to all interfaces).
 
Update: I just downloaded a FreeBSD VirtualBox image and tried the same thing on a different network. (Mac Desktop => VirtualBox => FreeBSD host => jail created by the same command). The jail can see the internet. Unlike my server this host (inside the VirtualBox) uses DHCP. However, I think it might be the hosting provider that doesn’t allow the traffic from em0/192.168.1.244.

I guess I have to set up a full NAT between my host (the server) and the jail. Is there an easier solution? I tried to use the same IP (199.199.199.199) for the jail on em0. This made me locked out from the server, even http wasn’t reachable.
 
Yes, you need to set up NAT on the host so that the 192.168.1.* addresses used by the jail(s) are hidden behind the host's public IP address (the 199.199.199.199 address). The other option would be to tell the upstream router about the 192.168.1.* addresses but that may not be possible if your ISP has set up the router so that only they can modify its settings.

I would change the setup a bit if I was you. I would create an lo1 interface by cloning (cloned_interfaces in rc.conf(5)) the lo(4) interface and then set the 192.168.1.244 address to the cloned lo1 interface. This would keep a better separation of the jail and the real host and it would be easier to write firewall rules for proper isolation.

One other thing. Disable all unnecessary services in the jail including sshd(8), you can control the jail very nicely from the real host and there's really no need to have SSH access directly to the jail.
 
Thank you. I’ll try to apply the rules from here: https://www.freebsd.org/doc/handbook/firewalls-ipfw.html

However, I am not sure what to do with the
Code:
defaultrouter="199.199.199.197"
in the host’s /etc/rc.conf. Is it okay to leave it there? Wouldn’t the lo1 try to use that route as well? (Which won’t work.) Or should I add something similar in the jail’s /etc/rc.conf?

As for disabling the sshd in the jail, I really need it. The host is a single user server (for me), however, a friend of mine uses it too for their php website. I don’t use php at all, however, safe_mode and base_dir doesn’t provide enough security. I could disable maybe twenty function to make it safer (exec, popen and so), however, they also need to run scripts (symfony2 init and capistrano deployment). This is how I ended up with the jail. I have never used it before. The idea is that they can do whatever they want inside the jail and I don’t have to read the php sources.

In the past they had only a chrooted sftp access. It’s neither safe for the host nor enough for their recent needs.
 
The jail doesn't have its own networking stack so whatever traffic is originated from the jail still uses the host's routing table. If you set up NAT properly the default router setting you now have is completely fine because the private range addresses used by the jail will not be visible outside the real host.
 
Well, almost everything works now.

/etc/rc.conf
Code:
# Firewall
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

# Ezjail
ezjail_enable="YES"
cloned_interfaces="${cloned_interfaces} lo1"
gateway_enable="YES"# enables the gateway
natd_enable="YES"# enables NAT
natd_interface="em0"# specify interface name of NIC attached to Internet
natd_flags="-dynamic -m -redirect_port tcp 192.168.1.244:22 2022"# -m = preserve port numbers; additional options are listed in natd(8)

/usr/local/etc/ipfw.rules
Code:
#!/bin/sh
ipfw -q -f flush
cmd="ipfw -q add"
skip="skipto 500"
pif="em0"
ks="keep-state"
good_tcpo="22,25,37,53,80,443,110"

$cmd 005 allow all from any to any via xl0  # Exclude LAN traffic
$cmd 010 allow all from any to any via lo0  # Exclude loopback traffic
$cmd 015 allow all from any to any via lo1  # Exclude loopback traffic

$cmd 100 divert natd ip from any to any in via $pif # NAT any inbound packets

# Allow the packet through if it has an existing entry in the dynamic rules table
$cmd 101 check-state

# Authorized outbound packets
$cmd 120 $skip udp from any to any 53 out via $pif $ks
$cmd 121 $skip udp from any to any 67 out via $pif $ks
$cmd 125 $skip tcp from any to any $good_tcpo out via $pif setup $ks
$cmd 130 $skip icmp from any to any out via $pif $ks

# Inbound
# Deny all inbound traffic from non-routable reserved address spaces
$cmd 00300 deny all from 192.168.0.0/16 to any in via $pif     #RFC 1918 private IP
$cmd 00301 deny all from 172.16.0.0/12 to any in via $pif      #RFC 1918 private IP
$cmd 00302 deny all from 10.0.0.0/8 to any in via $pif         #RFC 1918 private IP
$cmd 00303 deny all from 127.0.0.0/8 to any in via $pif        #loopback
$cmd 00304 deny all from 0.0.0.0/8 to any in via $pif          #loopback
$cmd 00305 deny all from 169.254.0.0/16 to any in via $pif     #DHCP auto-config
$cmd 00306 deny all from 192.0.2.0/24 to any in via $pif       #reserved for docs
$cmd 00307 deny all from 204.152.64.0/23 to any in via $pif    #Sun cluster interconnect
$cmd 00308 deny all from 224.0.0.0/3 to any in via $pif        #Class D & E multicast

# Deny public pings
$cmd 00310 deny icmp from any to any in via $pif

# Deny ident
$cmd 00315 deny tcp from any to any 113 in via $pif

# Deny all Netbios services.
$cmd 00320 deny tcp from any to any 137 in via $pif
$cmd 00321 deny tcp from any to any 138 in via $pif
$cmd 00322 deny tcp from any to any 139 in via $pif
$cmd 00323 deny tcp from any to any 81 in via $pif

# Deny fragments

$cmd 00330 deny all from any to any frag in via $pif

# Deny ACK packets that did not match the dynamic rule table
$cmd 00332 deny tcp from any to any established in via $pif

# Allow

# Allow inbound SSH connections
$cmd 00400 allow tcp from any to me 22 in via $pif $ks # setup limit src-addr 2

# Allow HTTP connections to internal web server
$cmd 00410 allow tcp from any to me 80 in via $pif $ks # setup limit src-addr 2

# Allow mail
$cmd 00411 allow tcp from any to me 25,143,465,993 in via $pif $ks # setup limit src-addr 2

# Everything else is denied

$cmd 499 deny log all from any to any

# NAT
$cmd 500 divert natd ip from any to any out via $pif # skipto location for outbound stateful rules
$cmd 510 allow ip from any to any

ezjail-admin create oi 'lo1|192.168.1.244'

There are a few questions.
1. I am not sure about the 015 rule. Is that safe? It made me able to ssh from the host to the jail.
2. I changed the setup limit src-addr 2 to $ks since I don’t want to restrict the amount of the connections. Is using keep-state there okay?
3. Do I need those NetBIOS rules?
4. ssh server -p 2022 says connection refused. If I add another rule to allow incoming on 2022 then it just waits forever. What should I do to forward the 2022 port of the public ip to the 22 port of the lo1? This is the primary goal of the whole thing.

Thank you.
 
Everything works, even filtering some ports from the jail. Except the ssh port forward. It drives me crazy.

I’ve switched from natd to ipfw NAT, there is a new config. It didn’t help.

/usr/local/etc/ipfw.rules
Code:
#!/bin/sh
cmd="ipfw -q add"
skip="skipto 10000"
wan="em0"
lan="lo1"
ks="keep-state"
oi_ip="192.168.1.244"

# Clear
ipfw -q -f flush

# Nat port forwarding
ipfw -q nat 1 config if $wan unreg_only same_ports redirect_port tcp $oi_ip:10077 10077

# Deny oi SSH and Postgresql
$cmd 5 deny ip from $oi_ip to any 22,5432

# Allow everything within the LAN
$cmd 10 allow ip from any to any via lo0
$cmd 20 allow ip from any to any via $lan

# Antispoof
$cmd 90 deny ip from any to any not antispoof in

$cmd 100 nat 1 ip from any to any via $wan in

# Allow stateful
$cmd 101 check-state

# Rules for outgoing traffic - allow everything that is not explicitely denied

# Allow all other outgoing connections

$cmd 300 $skip tcp from any to any via $wan out setup keep-state
$cmd 300 $skip udp from any to any via $wan out keep-state

# Allow ports

# Allow inbound SSH connections
$cmd 400 allow tcp from any to me 22 in via $wan setup $ks

# Allow HTTP connections to internal web server
$cmd 410 allow tcp from any to me 80 in via $wan setup $ks

# Allow BIND connections
$cmd 411 allow tcp from any to me 53 in via $wan setup $ks
$cmd 412 allow udp from any to me 53 in via $wan $ks

# Allow mail
$cmd 413 allow tcp from any to me 25,143,465,587,993 in via $wan setup $ks

# Allow jails oi ssh
$cmd 420 allow tcp from any to any 10077 in setup $ks

# Deny the rest

# Catch all other tcp/udp packets, but don't touch other packets, e.g. gre, esp, icmp
$cmd 9998 deny log tcp from any to any via $wan
$cmd 9999 deny log udp from any to any via $wan

# Allow anything after skipto
$cmd 10000 nat 1 ip from any to any via $wan out

$cmd 65534 allow ip from any to any

The oi jail has only lo1. I can ssh 192.168.1.244 -p 10077 on the host. It doesn’t work from outside on the public IP. I can’t see anything in the log. Any other port gets logged on deny so it’s not the IPS.

This config is almost the same as: https://gist.github.com/nileshgr/5990712

Any help or advice is highly appreciated.
 
Someone please help. I have already spent five days on fixing one single thing and it’s just crazy. I googled everything even Russian pages.

The goal is: forward 21211 port to the jail’s 21211 port. (ssh)

The problem is:
* If I set sysctl net.inet.ip.fw.one_pass to 0 then the port forward does not work. Everything else is fine.
* If I set it to 1 then the port forward works but all my ports are open and anyone can connect.
* If I set it to 0 and add deny_in then "everything" works except that I can’t make new connections to the host except the forwarded port. It even works without the rule 491.

I tried to add log everywhere but in case of net.inet.ip.fw.one_pass=0 the port forward does not appear in the log so I have no idea what to fix.

/etc/rc.cond.local
Code:
# Ezjail
ezjail_enable="YES"
cloned_interfaces="${cloned_interfaces} lo1"
gateway_enable="YES"        # Enables the gateway
firewall_nat_enable="YES"    # Enables NAT in ipfw
/usr/local/etc/ipfw.rules
Code:
#!/bin/sh

# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
cmd="ipfw -q add"
wan="em0"     # interface name of NIC attached to Internet
lan="lo1"     # Jail oi
skip="skipto 500"
ks="keep-state"
oi_ip="192.168.1.244"

# NAt port forward
ipfw -q nat 1 config log if $wan reset unreg_only same_ports redirect_port tcp 192.168.1.244:21211 21211

# Deny oi SSH and Postgresql
$cmd 00003 deny ip from $oi_ip to any 22,5432

# Change xl0 to LAN NIC interface name
$cmd 00005 allow all from any to any via xl0

# No restrictions on Loopback Interface
$cmd 00010 allow all from any to any via lo0
#$cmd 00010 allow all from any to any via lo1

# NAT
$cmd 00100 nat 1 ip from any to any via $wan in

$cmd 00101 check-state

# Allow any out
$cmd 00110 $skip tcp from any to any out via $wan setup $ks
$cmd 00111 $skip udp from any to any out via $wan $ks

# Allow outbound ping
$cmd 00250 allow icmp from any to any out via $wan $ks

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 00300 deny all from 192.168.0.0/16 to any in via $wan     #RFC 1918 private IP
$cmd 00301 deny all from 172.16.0.0/12 to any in via $wan      #RFC 1918 private IP
$cmd 00302 deny all from 10.0.0.0/8 to any in via $wan         #RFC 1918 private IP
$cmd 00303 deny all from 127.0.0.0/8 to any in via $wan        #loopback
$cmd 00304 deny all from 0.0.0.0/8 to any in via $wan          #loopback
$cmd 00305 deny all from 169.254.0.0/16 to any in via $wan     #DHCP auto-config
$cmd 00306 deny all from 192.0.2.0/24 to any in via $wan       #reserved for docs
$cmd 00307 deny all from 204.152.64.0/23 to any in via $wan    #Sun cluster interconnect
$cmd 00308 deny all from 224.0.0.0/3 to any in via $wan        #Class D & E multicast

# Allow public pings
$cmd 00310 allow icmp from any to any in via $wan

# Deny ident
$cmd 00315 deny tcp from any to any 113 in via $wan

# Deny all Netbios services.
$cmd 00320 deny tcp from any to any 137 in via $wan
$cmd 00321 deny tcp from any to any 138 in via $wan
$cmd 00322 deny tcp from any to any 139 in via $wan
$cmd 00323 deny tcp from any to any 81 in via $wan

# Deny fragments
$cmd 00330 deny all from any to any frag in via $wan

# Deny ACK packets that did not match the dynamic rule table
$cmd 00332 deny tcp from any to any established in via $wan

# Allow inbound ports

# Allow HTTP connections to internal web server
$cmd 00400 allow tcp from any to me 80 in via $wan setup limit src-addr 200

# Allow inbound SSH connections
$cmd 00410 allow tcp from any to me 22 in via $wan setup limit src-addr 200

# Allow BIND connections
$cmd 00411 allow tcp from any to me 53 in via $wan setup limit src-addr 200
$cmd 00412 allow udp from any to me 53 in via $wan $ks

# Allow mail
$cmd 00413 allow tcp from any to me 25,143,465,587,993 in via $wan setup limit src-addr 200


# Allow inbound oi SSH connections
$cmd 00491 allow tcp from any to me 21211 in via $wan setup limit src-addr 200
#$cmd 00491 allow tcp from any to any 21211 in via $lan setup limit src-addr 200

# Reject and log all other incoming connections
$cmd 00499 deny log all from any to any in via $wan

# NAT
$cmd 00500 nat 1 ip from any to any via $wan out # skipto location for outbound stateful rules
$cmd 00510 allow ip from any to any

# Everything else is denied and logged
$cmd 00999 deny log all from any to any
 
It seems like the firewall should be able to block ports. Restrictive firewall, then open the ports and ips you want. [a little late response]
 
Back
Top