HOWTO: Setup a jailed AppServer (Webserver/Mail/etc..) with ezjail

Slade

New Member

Reaction score: 9
Messages: 19

There are other ways you can do this but the following instructions I know work and should get you up and running. I spent a lot of time myself trying different solutions and this is the most portable/bulletproof solution that should work with pretty much any setup. This is the sort of thing I wish I had found when I was setting up jails.

The following includes using ezjail to set things up but it's not required, I've simply included it for people that may be starting from scratch (although it makes deploying jails in the future take about 30 seconds once you know what you're doing). Following these instructions you should be able to get up and running in 5 - 10 minutes. I've included what to add to your config files but you can start all these things manually from the shell without a reboot. Even though we're using ezjail I highly recommend reading the handbook section on jails so you have a solid understanding of what's going on.

External Resources:

FreeBSD Jails Handbook
ezjail website

Step 1.
Installing ezjail

Code:
cd /usr/ports/sysutils/ezjail/ && make install clean
man ezjail-admin at the shell prompt to get a basic understanding of the commands
Now we're going to install the basejail that all jails you create will use.

Code:
ezjail-admin install
Note: You can add a -P flag to have the ports collection installed in the basejail for all your jails to use. You can add the ports collection to the basejail at a later time also. The -P flag will also later update the ports collection of the basejail using portsnap.

Step 2.
Next we'll create the jail for our webserver.

Code:
ezjail-admin create WEBSERVER 10.1.1.1

Step 3.
Add the following to your hosts rc.conf (or manually via ifconfig)
ee /etc/rc.conf

Code:
#Setup interface all jails will use
#Make sure this netmask is unique in your rc.conf
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.1.1.1 netmask 255.255.255.0"

#Future jails can look like the following, aliases should always use 255.255.255.255 netmask
#ifconfig_lo1_alias0="inet 10.1.1.2 netmask 255.255.255.255"

# Enable port forwarding and packet filtering
pf_enable="YES"
pf_rules="/etc/pf.conf"

# Jails
ezjail_enable="YES"
To add your jails IP to a cloned loopback device via the shell enter the following at the shell prompt (this is not required if you plan on rebooting after all your configuration files are setup). Also unless you have your rc.conf setup this will not persist through a reboot.

Code:
ifconfig lo1 create
ifconfig lo1 inet 10.1.1.1 netmask 255.255.255.0
Step 4.
Setup PF NAT, change ext_if to reflect what your interface is (check via ifconfig).

ee /etc/pf.conf

Code:
#INTERFACES
ext_if="em0"
int_if = "lo1"
jailnet = $int_if:network

# NAT
nat on $ext_if from $jailnet to any -> ($ext_if)
Some useful PF commands to check to make sure everything is working and setup correctly:

Code:
pfctl -e                        Enable PF
pfctl -vnf /etc/pf.conf         Check /etc/pf.conf for errors, but do not load ruleset
pfctl -F all -f /etc/pf.conf    Flush all rules (nat, filter, state, table, etc.) and reload from the file /etc/pf.conf
If you haven't setup your network device via the shell prompt you now need to reboot so FreeBSD can load all your changed configuration files.

Step 5.
Once all the proceeding is running we can enter the jail.

Code:
ezjail-admin console WEBSERVER
Setup the jails resolv.conf (you can use different nameservers here). Your hosts /etc/resolv.conf can be copied to your jail if you don't want to use the google nameservers.

ee /etc/resolv.conf (inside the jail, note resolv doesn't have a trailing "e")

Code:
# google nameservers
nameserver 8.8.8.8
nameserver 8.8.4.4
Your jail should have network access now. If not type exit to return to the hosts shell and make sure your NAT rules are loaded.

Code:
pfctl -s nat
If no rules are displayed it means that either PF isn't enabled or there is an error in your pf.conf. At the shell prompt type "pfctl -vnf /etc/pf.conf" to check your pf.conf for errors. If there are no errors make sure PF is enabled "pfctl -e". You should also remove any blocking rules from your pf.conf to ensure that isn't causing a problem.

Note: You can't ping from inside jails because raw sockets are disabled. If dig and whois work then your jails network access is working. If you need to use ping you can change the following setting on your host.

Code:
sysctl security.jail.allow_raw_sockets=1
You will need to restart your jail for the changes to take affect. Make sure to disable raw sockets once you are done testing.

Code:
sysctl security.jail.allow_raw_sockets=0

Step 6.
Now that all the basics are setup we can add a redirect for incoming traffic. This will redirect port 80 on the host system to the jail which is running the webserver (you can change these ports to match whatever application you are running in your jail). Your full /etc/pf.conf on the host should now look something like this:

Code:
#INTERFACES
ext_if="em0"
int_if = "lo1"
jailnet = $int_if:network

# Name and IP of jails
WEBSERVER="10.1.1.1"

# NAT
nat on $ext_if from $jailnet to any -> ($ext_if)

# Redirect any packets requesting port 80 or 443 to jailed webserver
rdr pass on $ext_if inet proto tcp to port http -> $WEBSERVER port http
rdr pass on $ext_if inet proto tcp to port https -> $WEBSERVER port https

Step 7.
With everything up and running we can further tweak the jail. Check the timezone the jail is using via the date command. The jail should be synchronized to the hosts clock but the timezone may differ. If the timezone is incorrect simply use tzsetup in the jails shell to select the correct timezone.

Your JAILS /etc/rc.conf can look something like this:

Code:
rpcbind_enable="NO"             # Disable the RPC daemon
cron_flags="$cron_flags -J 15"  # Prevent lots of jails running cron jobs at the same time
syslogd_flags="-ss"             # Disable syslogd listening for incoming connections
sendmail_enable="NONE"          # Completely disable sendmail
clear_tmp_enable="YES"          # Clear /tmp at startup

Finishing Up
You should now be able to go back and add jails and services that you want very easily. Take a look at ezjail flavours to tailor a jails initial setup to your needs. Also once you have things setup you can edit your hosts pf.conf to actually block certain traffic and test to make sure that works accordingly. But both of those things are beyond the scope of this HowTo though.

I hope this helps people out.
 

indigo

New Member


Messages: 2

I've tried the same setup, but I've found it unnecessary to use gateway_enable="YES". It seems that pf doesn't seem to care about net.inet.ip.forwarding... any idea as to why?
 

indigo

New Member


Messages: 2

Never mind; this is due to the fact that packets are being forwarded to a local interface, not another external one.
 
OP
OP
S

Slade

New Member

Reaction score: 9
Messages: 19

Ah I didn't realize that. The way it sounded reading the handbook and other information was that any use of NAT required that. But I have removed gateway_enable="YES" and everything still works.
 

cleroy61

New Member


Messages: 14

Hello,

When I create a jail with ezjai, I can not access usr/ports from my jail to install nginx;

[cmd=]jexec 2 cd /usr/ports/www/nginx && make install clean[/cmd]
Code:
No such file or directory
It does not exist in fact, how can I link to my jail?
I’m out of my jail and next I tried a [cmd=]portsnap extract[/cmd] command
and then an [cmd=]portsnap fetch update[/cmd] command, thinking I did not open the softwares worn.

In fact, the /usr/ports/ directory is present but with the letter ‘l’ in front, preventing me from going there as a directory:

Code:
lrwxr-xr-x 1 root wheel 19 Sep 5 11:12 ports -> /basejail/usr/ports
Can You help me ?

Thanking you Sincerely Christophe
 

Nukama

Active Member

Reaction score: 43
Messages: 129

Welcome Christophe.

Ezjail links the jails /usr/ports to /basejail/usr/ports (ls(): l stands for symbolic link, d for directory).
/usr/jails/basejail is mounted on jail start to /usr/jail/${NAME.tld}/basejail (defined in /etc/fstab.${NAME_tld}).

Is your ${NAME.tld} jail running? $ [pman]ezjail-admin[/pman] list or $ [man]jls[/man].

Is your /usr/jails/basejail mounted on /usr/jails/${NAME.tld}/basejail?
$ [man]mount[/man] | grep basejail

Have you installed the ports-collection in /usr/jails/basejail?
# ezjail-admin install -P or # [man]portsnap[/man] -p /usr/jails/basejail/usr/ports/ fetch extract for installing the ports-collection and# portsnap -p /usr/jails/basejail/usr/ports/ fetch update for keeping it up to date.

Have you tried to use the commands within # ezjail-admin console ${NAME.tld}?

P.S.: Try to format your next post according to http://forums.freebsd.org/showthread.php?t=8816, thanks.
 

cleroy61

New Member


Messages: 14

Thanks for your answer(s) ;

I was in the market ;


So, I did not say, but i tried jls & ezjail-admin list :

Code:
# ezjail-admin list
STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
DR  1    10.1.13.6       takinium                       /usr/jails/takinium
I could connect to my jail takinium

[cmd=]# ezjail-admin console takinium[/cmd]

In my host :

[cmd=]# cd /usr/jails/takinium/basejail/[/cmd]

The mount :

Code:
# mount | grep basejail
/usr/jails/basejail on /usr/jails/takinium/basejail (nullfs, local, read-only)
in the host, I tried with success :

[cmd=]jexec 1 hostname[/cmd]

and without success :

[cmd=]jexec 1 cd /usr/ports/www/nginx && make install clean[/cmd]

[cmd=]ee /etc/rc.conf[/cmd] , there is
Code:
ezjail_enable="YES"
I tried with this command you suggested, it is the solution :
[cmd=]# portsnap -p /usr/jails/basejail/usr/ports/ fetch extract[/cmd]


Because when I logged with :

[cmd=]# ezjail-admin console takinium[/cmd]

I executed
Code:
takinium#cd /usr/ports/www/nginx 
make install clean
installation is running

in the begining of FreeBSD installation , I used to think that was enough :

[cmd=]portsnap fetch extract[/cmd]

the [cmd=]jexec 1 cd /usr/ports/www/nginx && make install clean[/cmd] command isn't running

thanking you sincerely
christophe
 

Nukama

Active Member

Reaction score: 43
Messages: 129

jexec() runs the command inside jails / directory. And IMHO jexec isn't able to run batched commands, at least it is not documented in jexec()'s manual page.

If you run # jexec 1 cd /usr && jexec 1 ls you can notice its behavior.

Your command # jexec 1 cd /usr/ports/www/nginx && make install clean would change into /usr/ports/www/nginx and run the make command inside your working directory of your host system.

make can be specified to run in a different directory, read the make()-manpage for further information.
But I advise against it, you should stick to the working solution and do your work inside the # [pman]ezjail-admin[/pman] console ${JAILNAME} or run a shell with # jexec -U username ${JID} /bin/csh.
 

Wiedmann

Member

Reaction score: 7
Messages: 29

Code:
jexec 1 cd /usr/ports/www/nginx && make install clean
You should use "ezjail-admin console" with the "-e" switch. Personally I use "portinstall foo" instead of "cd foo && make install clean":
Code:
ezjail-admin console -e 'portinstall nginx' takinium
BTW:
After a "portsnap fetch update" in the base system, you should also do a "portsdb -u".
 

UNIXgod

Daemon

Reaction score: 200
Messages: 1,088

Wiedmann said:
You should use "ezjail-admin console" with the "-e" switch. Personally I use "portinstall foo" instead of "cd foo && make install clean":
Code:
ezjail-admin console -e 'portinstall nginx' takinium
jexec is a system tool. if scripted it would be portable across multiple freebsd computers without the need for installation of ezjail on each machine which is simply a script itself using jexec while hiding the details of jexec.

For add hoc command execution using one over the other doesn't make a difference.
 

cleroy61

New Member


Messages: 14

I come back to you because I have another pb following the tutorial below. when I enter in my jail WEBSERVER, I can't see my tree jail :

Code:
WEBSERVER# ls -l
total 10
-rw-r--r--  2 root  wheel  798 Sep 15 05:43 .cshrc
-rw-------  1 root  wheel  486 Sep 15 11:20 .history
-rw-r--r--  1 root  wheel  155 Sep 15 05:43 .k5login
-rw-r--r--  1 root  wheel  303 Sep 15 05:43 .login
-rw-r--r--  2 root  wheel  265 Sep 15 05:43 .profile
Code:
digi00635# jls
   JID  IP Address      Hostname                      Path
     1  10.1.2.1        takinium                      /usr/jails/takinium
     2  10.1.1.1        WEBSERVER                     /usr/jails/WEBSERVER
What information do you need to help me?
 

jaxsin

New Member


Messages: 7

I am new and would like a little clarification on the original post. It seems like you are setting up a gateway with the
Code:
ifconfig_lo1="inet 10.1.1.1 netmask 255.255.255.0
I do not understand why you would want to put a jail service on this gateway while all other jails use the following code
Code:
#ifconfig_lo1_alias0="inet 10.1.1.2 netmask 255.255.255.255"
. What is the thinking behind this?

Code:
#Setup interface all jails will use
#Make sure this netmask is unique in your rc.conf
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.1.1.1 netmask 255.255.255.0"

#Future jails can look like the following, aliases should always use 255.255.255.255 netmask
#ifconfig_lo1_alias0="inet 10.1.1.2 netmask 255.255.255.255"
 

kpa

Beastie's Twin

Reaction score: 1,809
Messages: 6,318

FreeBSD requires that all the IPv4 addresses on a given network interface are non-conflicting in a way that either they are all from different non-overlapping subnets or if they are from overlapping or same subnets only one of the addresses in each subnet is configured with the proper net mask and other addresses of the subnet are configured with a 255.255.255.255 net mask. In this case the 101.1.1. is the first address from the 10.1.1.0/24 (in CIDR notation) subnet and the 10.1.1.2 address is the second address from the same subnet and that's why it has to be configured with the 255.255.255.255 net mask.

I think the technical reason for this limitation is that assigning an address on an interface with ifconfig(8) always adds a route to the routing table for the connected subnet. If there are overlapping entries in the routing table it's not possible to decide what to do with traffic going to an address that is in the overlapping address space. Using the 255.255.255.255 net mask solves the problem because there is only a single address in the subnet and it's unambiguous where to send the traffic going to that address.
 

cconrad

New Member


Messages: 3

I realize this is an old thread, but I wanted to say thanks to the OP for the helpful information (with a few modifications for my use case).

By the way, the line
Code:
pf_rules="/etc/pf.conf"
is not needed in /etc/rc.conf, because that is the default location, where pf looks for the configuration anyway.
 

andrewm659

Active Member

Reaction score: 4
Messages: 131

So I ran through this line by line and now I can't ssh into the FreeBSD 10.1 VM. I don't have gateway_enable="YES" in my rc.conf.
 

attilagyorffy

New Member

Reaction score: 1
Messages: 14

Slade I just wanted to say thank you for this write up. Using the steps above I now managed to finally get networking running within jails. Thanks again.
 
Top