A jail that is accessible from network without port forwarding

Balon

Balon

Hello! I am using FreeBSD 7.1 and on my server I have two different jails. One for database and second one for game server. What I need to know is if it is possible to assign game server's jail to public IP so I do not need to forward every port I need?

At the moment I'm using PF (Packet Filter) to make NAT for jails and to forward ports to all kinds of services I'm running in game server's jail. The problem is that it seems that PF is doing something nasty with packets I redirect to the game jail and players can login but they can't do anything else (game server is separated in few subservers - each handles different map in game). I've tried to run the game server on virtualized FreeBSD on my computer and everything worked. The only difference between OS on my home machine and server is just Packet Filter. So I want to test everything on the server without the PF, but is impossible as game server runs in jail and it won't work without NAT and port redirection. On #freebsd (freenode) some guys told me that it is actually possible to have my jail to "share" public IP with host system without port forwarding.

The question is: is it? If yes, how?

Thanks for any help!
 
Sure you can.
First of all you should not be using firewall/nat to drive traffic to jail.

Each jail is intended to have its own unique public static ip address.

The host and all jails share the same network stack.
The jails can be assigned the same public ip address as the host.
You should already know how to assign a jail a ip address.

A jail will process any port the application needs that is running in that
jail, as long as those port numbers are not being used by the host or
some other jail.

So port 80 used by the web server jail means the host can not be running
apache listening on port 80 and no other jail can be using port 80.

As a general rule, games are designed to work from a host with a public ip
address. IE: NOT AS A JAIL OR PC ON A PRIVATE LAN. This requirement is
hard coded into the game software. In very rare cases you can some times
fake the game out with NAT and forwarding for a single game user.

If you want to run a game server where multiple players login from the public
internet you will need a second static public ip address that you assign to
the jail containing the game server. Then and only them will it will function
like you exspect it to.
 
fbsd1 said:
Sure you can.
First of all you should not be using firewall/nat to drive traffic to jail.

Each jail is intended to have its own unique public static ip address.
Click to expand...

Considering that even the official wiki that the developers use has jails setup with a nat I'm going to go with this is incorrect ( http://wiki.freebsd.org/AppserverJailsHOWTO ). Using jails and a nat is a very common practice and there isn't anything wrong with that, unless you can provide some documentation to back that up. There are servers with 100's of jails are you saying each one needs a routable public ip?

As far as PF a common problem is people don't set it to skip local interfaces.

set skip on lo

Further diagnoses would require you posting your pf.conf to see what settings might be causing a problem.
 
Slade said:
Using jails and a nat is a very common practice and there isn't anything wrong with that, unless you can provide some documentation to back that up.
Click to expand...
There's nothing wrong with using NAT but there's also no need. NAT is also not mentioned in the jail(8) man page. I've setup several jails and I've never set one up with NAT.
 
SirDice said:
There's nothing wrong with using NAT but there's also no need. NAT is also not mentioned in the jail(8) man page. I've setup several jails and I've never set one up with NAT.
Click to expand...

I've had issues with non routable ip's with jails in certain environments. I've used jails in diskless servers that connect to a SAN with DHCP. I've setup FreeBSD as a guest with Xen Paravirtualization and with hardware virtualization on a KVM/QEMU host. I've also used it with Virtualbox. I've yet to see a step by step guide that shows a 100% bulletproof setup guaranteed to work that doesn't use NAT. The guides I did try didn't work. When I've used NAT ( as I describe in http://forums.freebsd.org/showthread.php?t=16860 ) it just works. And it works in a wide array of setups. In fact I've yet to see it not work. However I have seen a lot of posts where people had a lot of problems getting jails to have network access without it. I know people have setup jails without NAT (and if you can get it to work without NAT that's great) but my point was saying it's incorrect to say you shouldn't ever use it.
 
Slade said:
However I have seen a lot of posts where people had a lot of problems getting jails to have network access without it.
Click to expand...
I've seen them too. And the biggest problem is that those people have no idea how TCP/IP and routing work.

Funnily enough, if I go to the butcher everybody seems to order meat. If you go to a support forum everybody will have problems. People that don't want meat don't go to the butcher. People that don't have a problem won't seek help on a support forum.

This works for me, no nat, no frills:
Code: 
jail_enable="NO"
jail_list="build"
jail_build_rootdir="/jail/j1"
jail_build_hostname="build.dicelan.home"
jail_build_ip="192.168.1.181"
jail_build_interface="rl0"
jail_build_mount_enable="YES"
jail_build_devfs_enable="YES"
jail_build_devfs_rules="jail"

jail_apache_rootdir="/jail/j2"
jail_apache_hostname="apache.dicelan.home"
jail_apache_ip="192.168.1.182"
jail_apache_interface="rl0"
jail_apache_devfs_enable="YES"
jail_apache_devfs_ruleset="devfs_myjail_rule"
jail_apache_mount_enable="YES"

The host runs on rl0 and 192.168.1.180.
 
Ok since I would like a clear explanation why this doesn't work let's go down this road. The server has an internet-routable public ip while the jail does not.

Create an alias per ( http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-virtual-hosts.html ). The netmask of the host ip is 255.255.255.252 so the jail netmask doesn't conflict.

Code: 
ifconfig em0 alias 10.1.1.1 netmask 255.255.255.0

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 52:54:00:27:23:18
        inet 174.136.XXX.XXX netmask 0xfffffffc broadcast 174.136.XXX.XXX
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

Create a jail with the ip we just aliased.

Code: 
ezjail-admin create NGINX 10.1.1.1

Check the configuration file.

ee /usr/local/etc/ezjail/NGINX

Code: 
export jail_NGINX_hostname="NGINX"
export jail_NGINX_ip="10.1.1.1"
export jail_NGINX_rootdir="/usr/jails/NGINX"
export jail_NGINX_exec_start="/bin/sh /etc/rc"
export jail_NGINX_exec_stop=""
export jail_NGINX_mount_enable="YES"
export jail_NGINX_devfs_enable="YES"
export jail_NGINX_devfs_ruleset="devfsrules_jail"
export jail_NGINX_procfs_enable="YES"
export jail_NGINX_fdescfs_enable="YES"
export jail_NGINX_image=""
export jail_NGINX_imagetype=""
export jail_NGINX_attachparams=""
export jail_NGINX_attachblocking=""
export jail_NGINX_forceblocking=""
export jail_NGINX_zfs_datasets=""
export jail_NGINX_cpuset=""
export jail_NGINX_fib=""

Now enter the jail and setup an /etc/resolv.conf. At this point the firewall on the host is completely disabled so not only is there no NAT but no blocking to test the jails network connectivity.

At this point dig, whois, all network activity does not work. I do see you have a line that specifies the interface so I went back and added.

jail_NGINX_interface="em0" to /usr/local/etc/ezjail/NGINX

Code: 
export jail_NGINX_hostname="NGINX"
export jail_NGINX_ip="10.1.1.1"
export jail_NGINX_interface="em0"
export jail_NGINX_rootdir="/usr/jails/NGINX"
export jail_NGINX_exec_start="/bin/sh /etc/rc"
export jail_NGINX_exec_stop=""
export jail_NGINX_mount_enable="YES"
export jail_NGINX_devfs_enable="YES"
export jail_NGINX_devfs_ruleset="devfsrules_jail"
export jail_NGINX_procfs_enable="YES"
export jail_NGINX_fdescfs_enable="YES"
export jail_NGINX_image=""
export jail_NGINX_imagetype=""
export jail_NGINX_attachparams=""
export jail_NGINX_attachblocking=""
export jail_NGINX_forceblocking=""
export jail_NGINX_zfs_datasets=""
export jail_NGINX_cpuset=""
export jail_NGINX_fib=""

Stop and start the jail and it still doesn't work. I can take this exact same jail put it on lo1 and use NAT and get it working right away. So if I don't need to use NAT what do I need to change to get this working.
 
Slade said:
Ok since I would like a clear explanation why this doesn't work let's go down this road. The server has an internet-routable public ip while the jail does not.
Click to expand...

C'mon amigo, you answered your own question. :)

10/8 subnets are not routable across the 'net, period. So of course you need NAT in your specific case.

If you had two public IPs (one as the primary, one as an alias) you would not need NAT.
 
The problem is you have people that aren't qualifying when they say you don't need NAT for a jail. In fact a few people have gone so far to say you don't understand jails if you use NAT. I've seen it on the forums and on the freebsd mailing lists. Even when people are saying they are using non routable ip's. As someone who freely admits they are learning, but has read a ton of books and documentation lately, when you get conflicting information like this it's very confusing. I can tell you from reading through the handbook, Absolute FreeBSD 2nd Edition, and The best of FreeBSD Basics this information isn't made clear. I initially assumed there was stuff going on behind the scenes where FreeBSD was handling the network traffic essentially being NAT'd to the actual routable ip on the NIC when it was aliased. Obviously this isn't the case but I continue to see people that run into this problem (and are still told not to use NAT). I've helped a number of people on the mailing list just this week to get things working.
 
Setting aside mailing list discussions (which I do not have the complete context on), I will say my observation has been that many FreeBSD sysadmins have above average - expert levels of networking knowledge.

I think there is more to the story than folks making blanket rules about using NAT with jails; there are likely communication problems on either/both parts, and assumptions that both parties know their TCP/IP, relevant RFCs & NAT/PAT frontwards and back. This is clearly not always the case.

I wouldn't sweat the issue any more, and it sounds like you understand now the situations for the "don't use NAT" recommendation. It simply doesn't apply in all cases. For those that it does apply, I tend to agree with it. (NAT is a necessary, but unpleasant, IPv4 kludge.)
 
You must log in or register to reply here.
Back
Top