Request for second opinion on my config for IPFW with 2 jailed servers (FreeBSD 8.1)

G

ghostcorps

Hi Guys,

Hopefully this request will help others with the same question, I haven't found any singular source with a complete answer.

I have undertaken quite a challenge for myself and I have been winning so far. But I am coming up to the hardest part and need another set of eyes/brains to look over the plan before I trash the system and/or lock myself out.

The setup is fairly simple, for this forum anyway! I have a remote server (no physical access at all) running FreeBSD 8.1, it is a very minimal install as it is simply the host for two jailed servers. One is a webserver, the other is the data server. At the moment, everything works a treat. But, there is no firewall... and we want one.

I have done alot of reading and have come up with a list of settings that I think will do the job. And one big question.

Ok, maybe two questions. Firstly, do I need natd? I'm certain that I do, but if someone says no, it will make things so much easier for me. Secondly, if I do need natd, do I need address redirection if I am already using port redirection?

Below are the modifications I propose to enable ipfw on the host to protect the system, please let me know if I am missing anything, or if I have something there that shouldn't be:


host# /etc/rc.conf
Code: 
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_type="simple"
gateway_enable="YES"
natd_enable="YES"
natd_interface="em0"
natd_flags="-f /etc/natd.conf"


webserver# /etc/rc.conf
Code: 
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_type="client"


dataserver# /etc/rc.conf
Code: 
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_type="client"


host# /etc/natd.conf
Code: 
redirect_port tcp 203.xxx.xxx.xxx:21 21
redirect_port tcp 203.xxx.xxx.xxx:80 80
redirect_port tcp 203.xxx.xxx.xxx:443 443
  • nb. 203.xxx.xxx.xxx is the external ip for the webserver.

host# /boot/boot.loader
Code: 
ipfw_load="YES"
ipdivert_load="YES"
net.inet.ip.fw.default_to_accept="1"   ##Safety net to be disabled after testing is complete


host# /etc/sysctl.conf
Code: 
net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=5


Options for KERNEL customizations
Code: 
options IPFIREWALL
options HZ=1000
options IPFIREWALL_DEFAULT_TO_ACCEPT ##Safety net to be disabled after testing is complete
options IPDIVERT

  • nb. I have not even begun worrying about the rules yet. I just want to know that this configuration will allow full access before I start locking it down.


With any luck I have put together a solid set of configurations to go from no firewall at all to ipfw with nat.


What is the public opinion? Am I on the right track?


Thanks for your time, I know it's not a simple one.
 
ghostcorps said:
host# /etc/natd.conf
Code: 
redirect_port tcp 203.xxx.xxx.xxx:21 21
redirect_port tcp 203.xxx.xxx.xxx:80 80
redirect_port tcp 203.xxx.xxx.xxx:443 443
  • nb. 203.xxx.xxx.xxx is the external ip for the webserver.
Click to expand...

Hello,

Is this ip address, the only external address of the host? Or you have multiple ip addresses and you gave that to jail?

What is your jail config? Is interface of jails em0 or virtual lan (vlan0). What is the ip addresses?

For example if you use vlan0 for jails, and your webservers ip address is 192.168.0.1, you must do like that:
Code: 
redirect_port tcp 192.168.0.1:80 80
The ip addrses should belong to jail of webserver.
Code: 
redirect_port tcp jail_ip:jail_port host_port
 
Thanks for the reply Jailed, sorry it has taken so long to get back to you. I had to put this on the backburner while working out some other stuff.


To answer your question:

I have three external IPs. 203.xxx.xxx.x12 is the host, 203.xxx.xxx.x13 is the webserver address that I used in the natd.conf above and 203.xxx.xxx.x14 which is the dataserver, this should have no ports available externally.

There are no internal IPs set.

Here is my jail configuration as per rc.conf in the host:

host# /etc/rc.conf
Code: 
ifconfig_em0="inet 203.xxx.xxx.x12 netmask 255.255.255.248"
defaultrouter="203.xxx.xxx.x09"
inetd_enable="YES"
inetd_flags="-wW -a 203.xxx.xxx.x12"
ifconfig_em0_alias0="inet 203.xxx.xxx.x13 netmask 255.255.255.255"
ifconfig_em0_alias0="inet 203.xxx.xxx.x14 netmask 255.255.255.255"
syslogd_enable="YES"
syslogd_flags="-ss"
syslogd_flags="-a 203.xxx.xxx.x12"
syslogd_flags="-a 203.xxx.xxx.x13"
syslogd_flags="-a 203.xxx.xxx.x14"
# Jail general settings
jail_set_hostname_allow="NO"
jail_enable="YES"
jail_list="webserver database"
jail_interface="em0"
jail_devfs_enable="YES"
jail_procfs_enable="YES"
# settings per jail listed in jail_list
#Webserver
jail_webserver_rootdir="/usr/gaols/webserver"
jail_webserver_hostname="webserver"
jail_webserver_ip="203.xxx.xxx.x13"
jail_webserver_devfs_ruleset="devfsrules_jail"
#Database
jail_database_rootdir="/usr/gaols/database"
jail_database_hostname="database"
jail_database_ip="203.xxx.xxx.x14"
jail_database_devfs_ruleset="devfsrules_jail"
 
I found a small window where I could enable natd and risk complete lock-out, and was not locked out :)

I used the exact natd settings I described above, so I presume it is working so far :D

Now to enable the firewall...?
 
Hello,

NATd is needed for setups like this:
Say that you have an external IP 1.2.3.4 and you want to forward 1.2.3.4:80 to 1.2.3.5:80 or any a.b.c.d on other NICs, so you can use NATd.

You say that you have 3 external IPs.

Server = 203.xxx.xxx.x12
WWW = 203.xxx.xxx.x13

If you want to use 203.xxx.xxx.x12:80 for WWW and connect it to 203.xxx.xxx.x13, you should use NATd.

However, you can use 203.xxx.xxx.x13 itself for WWW without NATd configuration. It's unnecessary.

You can use 203.xxx.xxx.x12 for (e.g.) SSH connections to server, use 203.xxx.xxx.x13 for WWW and 203.xxx.xxx.x14 for DB without NAT.

Add
Code: 
jail_webserver_interface="em0"
and
Code: 
jail_database_interface="em0"

Stop aliasing IPs. It's unnecessary. Using this IPs with jail configuration with em0 interface will be enough.

You can use a firewall without NAT configuration. I suggest closed firewall. It will deny all connections by default. And you will set which connections to allow by firewall commands.

If you still want NATd configuration, I can show you some sample.

But, using firewall without NATd will secure your server. For example, you can close connections to your database server from outside net. You can only allow your WWW server to connect it. So any user from WAN won't access your DB although you use public IP.
 
Wow you have given me alot to do :)

Thanks for the thorough reply.

I will have to wait till Monday to get onto this, but if you don't mind I'll get back to you if I have any trouble.
 
If your server is located on a datacenter, and you control it remotely,

add this rules to your ipfw.rules
Code: 
/sbin/ipfw -q add 100 allow all from any to 203.xxx.xxx.x12
/sbin/ipfw -q add 101 allow all from 203.xxx.xxx.x12 to any

So it will allow all connections to and from 203.xxx.xxx.x12

This will help you not to lock yourself.

Add rules with IP specific.

This will allow all internal connections
Code: 
/sbin/ipfw -q add 10 allow all from me to me

This will allow all passive ports to all IPs.
Code: 
/sbin/ipfw -q add 50 allow all from any to any 1024-65535

This will allow all HTTP connections to your Web Server.
Code: 
/sbin/ipfw -q add 1000 allow all from any to 203.xxx.xxx.x13 80

Using a closed IPFW with this rules will secure other ports and IPs from WAN.
 
Thanks again!

I have to wait another day or so before I can make any more changes, but I wanted to confirm a couple things, if you don't mind?

Firstly, are these the lines you say are unnecessary after adding the interface lines?

host# /etc/rc.conf
Code: 
ifconfig_em0_alias0="inet 203.xxx.xxx.x13 netmask 255.255.255.255"
ifconfig_em0_alias0="inet 203.xxx.xxx.x14 netmask 255.255.255.255"


I was following the guides here which advised using NATd, but it makes sense that I would not need it if I am not actually translating any address':

http://forums.freebsd.org/showthread.php?p=123191#post123191
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html


Finally, the big question... after disabling NATd, adding the interface lines, removing the aliasing, and setting the firewall type to 'closed'... 'should' the configuration in my first post work with the rule set below to only allow; ssh to the host, www (http, https, ftp & rtmp) to the webserver and internal connections to the database?



#host /etc/ipfw.rules
Code: 
IPF="ipfw -q add"
ipfw -q -f flush

#loopback
$IPF 10 allow all from any to any via lo0
# allow all internal connections
$IPF 11 allow all from me to me
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
# allow all passive ports to all IPs.
$IPF 51 allow all from any to any 1024-65535
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any

# ensure remote access (temporary?)
$IPF 100 allow all from any to 203.xxx.xxx.x12
$IPF 101 allow all from 203.xxx.xxx.x12 to any

# deny and log everything
$IPF 500 deny log all from any to any

# Allow ssh to host
$IPF 900 allow all from any to 203.xxx.xxx.x12 22

# allow all HTTP connections to Web Server.
$IPF 1000 allow all from any to 203.xxx.xxx.x13 80
$IPF 1100 allow all from any to 203.xxx.xxx.x13 443
$IPF 1200 allow all from any to 203.xxx.xxx.x13 5080
$IPF 1300 allow all from any to 203.xxx.xxx.x13 21

Thanks for all your time, I am learning heaps :)
 
ghostcorps said:
Firstly, are these the lines you say are unnecessary after adding the interface lines?

host# /etc/rc.conf
Code: 
ifconfig_em0_alias0="inet 203.xxx.xxx.x13 netmask 255.255.255.255"
ifconfig_em0_alias0="inet 203.xxx.xxx.x14 netmask 255.255.255.255"
Click to expand...

Absolutely unnecessary. Configuring an IP address to a Jail is enough to assign that IP to server. Jail will automatically assign the IP address on the NIC interface and start to use this IP.

For example you have 8 IPs, You will use 1 of them for server and 2 of them for Jails, however you want other 5 unused IPs accessible, you can use IP aliasing. So this 5 aliased IPs will point to main server.

But in your situation, it's unneeded. You have jail configuration and aliasing IPs is not needed.

Also if you require, you can use multiple IPs for jails without aliasing.

You can separate multiple IPs with comma like this:
Code: 
jail_www_ip="1.2.3.4, 1.2.3.5, 1.2.3.6"

The NAT issue,

I'm using NAT in my server(s). I have about 15 server in my home office. There is 3 different internet connection.

I'm using NAT to connect the local servers to internet, and forward ports to servers. I can tell my gateway server to forward port 80 to the web server (not jail, physical server). That's port forwarding part of NAT. And I can tell my gateway server to allow some servers to connect internet over itself; and that's the Network Address Translation.

You have a unique physical server and jails inside it. It's not necessary to use NAT.

But if you are willful to use NAT, you can create a cloned interface (VLAN).

Code: 
vlan0 = 192.168.0.1 (192.168.0.0/24)
Jail 1 = 192.168.0.2
Jail 2 = 192.168.0.3
Jail 3 = 192.168.0.4

"vlan0" will be a virtual NIC interface in your server. It will act like that. So your jails will work on local IP addresses, not public ones.

Than you can use NAT to connect this Jails to internet, and forward ports to jails.

You can use NAT in this situation; but if you don't have a reason for that, it's still meaningless.

ghostcorps said:
Finally, the big question... after disabling NATd, adding the interface lines, removing the aliasing, and setting the firewall type to 'closed'... 'should' the configuration in my first post work with the rule set below to only allow; ssh to the host, www (http, https, ftp & rtmp) to the webserver and internal connections to the database?
Click to expand...

Yes you can only allow whatever you want, or deny whatever you want with IPFW without NATd.

I will repeat my question again, is this server on a datacenter and do you have only remote control? Or do you have physical access?

I will help on configurations and comment on ruleset, according to your access type. ;)

Have a good day!
 
I will keep things simple and do without NAT for now.

I customised my kernel with the options in the first post and it seems to work so far.


jailed said:
I will repeat my question again, is this server on a datacenter and do you have only remote control? Or do you have physical access?
Click to expand...

The server is not physically accessible, I am in Australia and it is in the US.

I login to the host via ssh, from there I use jexec to 'login' from root to the jailed servers. The host has a fall-over page also on port 80 in case apache crashes on the webserver.

I can 'login' to the database server but MySQL is only accessible from the webserver via port 3306.


Thankyou for your patience :)
 
Hello again

I have just enabled my firewall with the settings below, but while I can ssh into the host, I can not access the webserver via ftp or www, what have I done wrong?

Jailed: I don't think I was clear enough? The server is in a datacentre in the US, while I am in Australia.


HOST #/etc/rc.conf
Code: 
hostname="host.HOSTNAME"
sshd_enable="YES"
ifconfig_em0="inet xxx.xxx.xxx.012 netmask 255.255.255.248"
defaultrouter="xxx.xxx.xxx.009"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_type="closed"
apache22_enable="YES"
apache22_ssl_enable="YES"
apache22_http_accept_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a xxx.xxx.xxx.012"
rpcbind_enable="NO"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
syslogd_enable="YES"
syslogd_flags="-ss"
syslogd_flags="-a xxx.xxx.xxx.x12"
syslogd_flags="-a xxx.xxx.xxx.x13"
syslogd_flags="-a xxx.xxx.xxx.x14"
# Jail general settings
jail_set_hostname_allow="NO"
jail_enable="YES"
jail_list="webserver database"
jail_interface="em0"
jail_devfs_enable="YES"
jail_procfs_enable="YES"
# Webserver
jail_webserver_rootdir="/usr/gaols/webserver"
jail_webserver_interface="em0"
jail_webserver_hostname="webserver"
jail_webserver_ip="1xxx.xxx.xxx.x13"
jail_webserver_devfs_ruleset="devfsrules_jail"
# database
jail_database_rootdir="/usr/gaols/database"
jail_database_interface="em0"
jail_database_hostname="database"
jail_database_ip="xxx.xxx.xxx.x14"
jail_database_devfs_ruleset="devfsrules_jail"

HOST #/etc/ipfw.rules
Code: 
IPF="ipfw -q add"
ipfw -q -f flush

# loopback
$IPF 10 allow all from any to any via lo0

# allow all internal connections
$IPF 11 allow all from me to me
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
# allow all passive ports to all IPs.
$IPF 51 allow all from any to any 1024-65535
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any

# ensure remote access (temporary?)
$IPF 100 allow all from any to xxx.xxx.xxx.x12
$IPF 101 allow all from xxx.xxx.xxx.x12 to any

# deny and log everything
$IPF 500 deny log all from any to any

# Allow ssh to host
$IPF 900 allow all from any to xxx.xxx.xxx.x12 22

# allow all HTTP connections to Web Server.
$IPF 1000 allow all from any to xxx.xxx.xxx.x13 80
$IPF 1100 allow all from any to xxx.xxx.xxx.x13 443
$IPF 1200 allow all from any to xxx.xxx.xxx.x13 5080
$IPF 1300 allow all from any to xxx.xxx.xxx.x13 21
$IPF 1300 allow all from any to xxx.xxx.xxx.x13 8088

HOST #/boot/loader.conf
Code: 
ipfw_load="YES"
ipdivert_load="YES"
net.inet.ip.fw.default_to_accept="1"   ##Safety net to be disabled after testing is complete

HOST #/etc/sysctl.conf
Code: 
net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=5

KERNEL customizations
Code: 
options IPFIREWALL
options HZ=1000
options IPFIREWALL_DEFAULT_TO_ACCEPT ##Safety net to be disabled after testing is complete
options IPDIVERT


WEBSERVER #/etc/rc.conf
Code: 
hostname="web.HOSTNAME"
ifconfig_em0="inet xxx.xxx.xxx.x13 netmask 255.255.255.255"
defaultrouter="1xxx.xxx.xxx.x09"
apache22_enable="YES"
apache22_ssl_enable="YES"
apache22_http_accept_enable="YES"
rpcbind_enable="NO"
clear_tmp_enable="YES"
sendmail_enable="NONE"
sshd_enable="NO"
inetd_enable="NO"
kern_securelevel_enable="YES"
kern_securelevel="3"
portmap_enable="NO"
icmp_drop_redirect="YES"
icmp_log_redirect="YES"
log_in_vain="YES"
# Enable Firewal
firewall_enable="YES"
firewall_type="client"
pureftpd_enable="YES"
red5_enable="YES"

DATASERVER #/etc/rc.conf
Code: 
hostname="IP.ADDRESS"
ifconfig_em0="inet xxx.xxx.xxx.x14 netmask 255.255.255.255"
defaultrouter="1xxx.xxx.xxx.x09"
apache22_enable="YES"
apache22_ssl_enable="YES"
apache22_http_accept_enable="YES"
rpcbind_enable="NO"
clear_tmp_enable="YES"
sendmail_enable="NONE"
sshd_enable="NO"
inetd_enable="NO"
kern_securelevel_enable="YES"
kern_securelevel="3"
portmap_enable="NO"
icmp_drop_redirect="YES"
icmp_log_redirect="YES"
log_in_vain="YES"
# Enable Firewal
firewall_enable="YES"
firewall_type="client"
pureftpd_enable="YES"
red5_enable="YES"
 
Hello,

I was very busy so I couldn't reply you.

Fist, I see that you set firewall configuration under jail. Why?

Then,

Please remark this line and try,

Code: 
# deny and log everything
# $IPF 500 deny log all from any to any

You deny all and then allow ports of jail on the following lines. (900-1300)
 
Thanks Jailed.

No need to apologize, I have not been in a hurry. I appreciate any time you can share with me :)

I can not figure out where I came up with the firewall rules in the jail rc.conf or the ones you suggest to have commented out. I will test the changes tomorrow and get back to you.

Thanks again for your time :)
 
You must log in or register to reply here.
Back
Top