VIMAGE: Setup Guide

Remington

Remington

Here is the script to automatically start VIMAGE and non-VIMAGE jails during boot-up.

This script was updated to work with FreeBSD 11 and it may work with older releases. Please do let me know if there are further improvements that can be made and I'll update this post. Thanks!

You must know how to do the following:
1) Create jail
2) Customize and re-compile GENERIC kernel
3) VIMAGE is highly experimental and its not ready for production. Do it at your own risk.

You will need to edit /etc/rc.conf to include the following lines to start jail at boot. It is recommended to use ezjail for jail management since it offers excellent tools such as create, destroy, update and console. It simplifies managing jails.

Code: 
#ezjail_enable="YES"    # This should be disabled and not supported under FreeBSD 11
jail_enable="YES"

I use FreeBSD's service jail [start|stop|restart] [jail_name] to start or stop jails at the command line.

Officially, ezjail does not support VIMAGE as its still highly experimental. You will need to add options VIMAGE to /usr/src/sys/amd64/conf/GENERIC and re-compile the kernel.

This needs to be added to /etc/rc.conf. Firewall is not required.
Code: 
# Network configuration
ifconfig_igb0="inet 192.168.1.100 netmask 255.255.255.0"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0"

# Jail
jail_enable="YES"
jail_list="jail1 jail2"

# Firewall - Enable this if want to use firewall inside host
#pf_enable="YES"
#pflog_enable="YES"

Packet filterings for bridge to be used with firewall. You can ignore this as it is not required.
/etc/sysctl.conf
Code: 
net.inet.ip.forwarding=1                   # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0              # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0              # Packet filter on the bridge interface
net.link.bridge.pfil_member=0              # Packet filter on the member interface
security.bsd.unprivileged_read_msgbuf=0
# This is only for routing tables if any
# (do not create default routing tables for all FIB's)
net.add_addr_allfibs=0

This is required for jail and VIMAGE to work correctly.
/etc/jail.conf
Code: 
# Jail configuration - /etc/jail.conf

allow.mount;
allow.raw_sockets     = "1";
allow.set_hostname    = "0";
allow.sysvipc         = "1";

host.hostname         = "${name}.local";
path                  = "/jails/${name}";
mount                 = "/jails/basejail /jails/${name}/basejail nullfs ro 0 0";  # Use this mount for ezJail's jails
#mount.fstab          = "/etc/fstab.${name}";                                     # Optional: mount external fstab files
mount.devfs;
mount.fdescfs;
mount.procfs;
devfs_ruleset         = "4";

exec.clean;
exec.start           += "/bin/sh /etc/rc";
exec.stop             = "/bin/sh /etc/rc.shutdown";
exec.consolelog       = "/var/log/jail_${name}_console.log";

# Jail with VIMAGE
jail1 {
  $if                 = "0";                  # Jail ID number
  $ip_addr            = "192.168.1.101";      # Jail ipv4 address
  $ip_route           = "192.168.1.100";      # Gateway or host's ipv4 address
  $ip6_addr           = "fe00::1";            # Jail ipv6 address
  $ip6_route          = "fe00::";             # Gateway or host's ipv6 address
  vnet;
  vnet.interface      = "epair${if}b";
  exec.prestart       = "ifconfig epair${if} create up";
  exec.prestart      += "ifconfig bridge0 addm epair${if}a";
  exec.start          = "/sbin/ifconfig epair${if}b inet ${ip_addr} up";
  exec.start         += "/sbin/ifconfig epair${if}b inet6 ${ip6_addr} up";
  exec.start         += "/sbin/route add default -gateway ${ip_route}";
  exec.start         += "/sbin/route add -inet6 default -gateway ${ip6_route}";
  exec.start         += "/bin/sh /etc/rc";
  exec.stop           = "/bin/sh /etc/rc.shutdown";
  exec.poststop       = "ifconfig epair${if}a destroy";
  persist;
}

# Jail without VIMAGE
jail2 {
   ip4.addr           = "192.168.1.102";  # IP4 address
   ip6.addr           = "fe00::1";        # IP6 address (optional)
   interface          = "igb0";           # Network interface
   mount             += "/mnt/media1  /jails/jail2/mnt/media1 nullfs rw 0 0";   # This is optional to add more mounts for jail
   mount             += "/mnt/media2  /jails/jail2/mnt/media2 nullfs rw 0 0";   # This is optional to add more mounts for jail
}

You can utilize firewall inside jail with VIMAGE but you will have to add the following lines to /etc/rc.conf inside jail. Use IPFW as its the only firewall that works well inside jail with VIMAGE. If you use different firewall such as PF inside jail and it will cause problems. That is why VIMAGE is highly experimental.
Code: 
# Jail Firewall - Enable this if want to use firewall inside jail
firewall_enable="YES"
firewall_type="open"


Revision History:
21/12/14 - Added 'route add default' and cleaned up the script
22/05/15 - Added firewall options to /etc/rc.conf
26/05/15 - Added ${path} to jail.conf
31/07/16 - Removed loopback interface. Added ipv6 support
26/09/16 - Updated jail.conf
20/10/16 - Updated the guide for FreeBSD 11
30/10/16 - Updated the mount point
 
Last edited:
Thanks heaps for this how to guide!

What would be a great addition to this is to explain what you are doing in regards to networking, maybe a networking diagram.

Also I notice you mention PF in the jail rc.conf but you seem to be enabling IPFW, you also do not provide a more complex networking setup other than firewall_type="open".

You clone the loopback on the host but do not seem to use it, is there a reason for this?

This tutorial seems to be focused on a virtual machine based FreeBSD host. What about VIMAGE jails (with private IP) running on a host with a single public IP and interface? How would the configuration differ in that scenario?
 
perkypork said:
Also I notice you mention PF in the jail rc.conf but you seem to be enabling IPFW, you also do not provide a more complex networking setup other than firewall_type="open".
Click to expand...

Firewall isn't required for either host or jails and both can be disabled.

You clone the loopback on the host but do not seem to use it, is there a reason for this?
Click to expand...

I removed the loopback as there is no effect.

This tutorial seems to be focused on a virtual machine based FreeBSD host. What about VIMAGE jails (with private IP) running on a host with a single public IP and interface? How would the configuration differ in that scenario?
Click to expand...

If the server only have one public ip address then private ip address may be needed with NAT or proxy and this is useful for multiple jails with websites. The only scenario for anyone using VIMAGE is to have its own network stack, different MAC address or firewall within Jail otherwise normal Jail is good enough.
 
12.0-beta3 is published and now vimage is included in the base kernel. It's no longer required to do a kernel compile to enable vimage. Also the pf firewall has been fixed to work inside of a vnet jail.

Have question about the bridge. The example shows one pair or epairs attached to the bridge. If having say 4 vnet jails can all of them be attached to the same bridge? Or does each vnet jail epair set need it's own bridge?

If the interface added as a member to the bridge is the host's interface facing the public internet and the vnet jail is running a firewall with NAT, would pings of a public ip address pass directly out that bridge interface or would the ping packet exit the vnet jail firewall and be handed off to the firewall running on the host and be NATed again? What is really happening here?

In the example, the hosts rc.conf you have ifconfig_igb0="inet 192.168.1.100 netmask 255.255.255.0" is this the interface facing the LAN? The jail1 vnet jail has a "/sbin/route add default -gateway ${ip_route}" command which is ip 192.168.1.100. Why is this done?

How would you test your vnet jail to verify it's working?
 
fbsd1 said:
Have question about the bridge. The example shows one pair or epairs attached to the bridge. If having say 4 vnet jails can all of them be attached to the same bridge? Or does each vnet jail epair set need it's own bridge?
Click to expand...

Each jail needs their own epair sets so jail start/shutdown/restart will work properly.

If the interface added as a member to the bridge is the host's interface facing the public internet and the vnet jail is running a firewall with NAT, would pings of a public ip address pass directly out that bridge interface or would the ping packet exit the vnet jail firewall and be handed off to the firewall running on the host and be NATed again? What is really happening here?
Click to expand...

All vnet jail traffic will pass through the bridge interface just like it has its own NIC bypassing host's NAT and firewall.

In the example, the hosts rc.conf you have ifconfig_igb0="inet 192.168.1.100 netmask 255.255.255.0" is this the interface facing the LAN? The jail1 vnet jail has a "/sbin/route add default -gateway ${ip_route}" command which is ip 192.168.1.100. Why is this done?
Click to expand...

It needs host's NIC IP address to pass in/out the traffic. Some servers have two NICs so there's two gateways.

How would you test your vnet jail to verify it's working?
Click to expand...

ssh.

I have not checked out the 12.0 beta release so I'm not sure if the above config is still the same.
 
Remington said:
Each jail needs their own epair sets so jail start/shutdown/restart will work properly.
Click to expand...

This answer does not reflect the question. The question as about bridges. If having say 4 vnet jails can all of them have their epair attached to the same bridge? Or does each vnet jail epair set need it's own bridge?

From your answers I can see how a ping from a vnet console would pass out to the public internet. How will the reply target the vnet jail?

Use ssh to test if networking is working for in/out of vnet jail. Need more info on how to do this.

I tested your config on 12.0 beta3 with a few changes. Removed exec.clean, allow.set_hostname from global area & persist; from jail1. Moved allow.raw_sockets = "1"; & allow.sysvipc = "1"; to jail2. Did not do any of the sysctl commands. In the vnet jail's rc.conf added statements to enable pf firewall & gateway_enable ="YES". I did not use the "service" command to start/stop the jails so no host rc.conf jail statements. I setup each jail in it's own jail.conf file and use the jail(8) native command for start/stop.
jail -f /usr/local/etc/jailname -c & jail -f /usr/local/etc/jailname -r jailname
The trailing -c means create/start the vnet jail & the trailing -r means remove/stop the vnet jail.

The jls -h -j jailname command shows the vnet jail is running and lists all it's options. I can jexec into the vnet jail's console and issue ifconfig and see epair#b ip address. Can not ping the host or the public internet. Vnet jail pf rules say pass log all. Nothing shows up in the vnet jail's pflog file. ps ax from the vnet console doesn't show the pflog running. Think this is a bug with pflog starting in a vnet jail.

ezjail doesn't use the jail.conf file. Its still stuck using the rc.conf method and gets a warning every time a jail is started. Jail.conf was introduced in 9.1, This is the 3rd major release and ezjail has not made the move to jail.conf. I use qjail which uses the jail.conf file and is a fork of ezjail with many added features, including vnet jails.

Thanks for the info, its been very useful so far.
 
Thank you for the instructive how-to, very useful for people like me trying to use VNET jails.

I have not checked out the 12.0 beta release so I'm not sure if the above config is still the same.
Click to expand...

Does this need any update for FreeBSD 13.0-RELEASE?
 
You must log in or register to reply here.
Back
Top