Best practice: jails

[nanotek]

I took @SirDice's advice and just reinstalled all necessary applications and transferred configuration files and user data to the new server, instead of making and restoring an image. In the process, I decided to change my jail setup a little. In the past, I would just set up jails on aliases on the primary interface, for example:

rc.conf

ifconfig_rl0="inet 10.0.0.50/24"
defaultrouter="10.0.0.150"
ifconfig_rl0_alias0="inet 10.0.0.110/24"  # srv
ifconfig_rl0_alias1="inet 10.0.0.120/24"  # mail
ifconfig_rl0_alias2="inet 10.0.0.100/24"  # relay
ifconfig_rl0_alias3="inet 10.0.0.130/24"  # rtd

ifconfig

ath0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 2290
        ether 00:16:e3:14:8f:33
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: IEEE 802.11 Wireless Ethernet autoselect (autoselect)
        status: no carrier
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2008<VLAN_MTU,WOL_MAGIC>
        ether 00:a0:d1:38:e6:f5
        inet 10.0.0.50 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::2a0:d1ff:fe38:e6f5%rl0 prefixlen 64 scopeid 0x2
        inet 10.0.0.110 netmask 0xffffffff broadcast 10.0.0.255
        inet 10.0.0.120 netmask 0xffffffff broadcast 10.0.0.255
        inet 10.0.0.100 netmask 0xffffffff broadcast 10.0.0.255
        inet 10.0.0.130 netmask 0xffffffff broadcast 10.0.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33184

The advantage of this first method being that no NAT is required by PF; it's all done at the router, which although very basic in function, is much simpler than messing around with pf.conf.

On the new system, I decided to create a network on a cloned device for my jails:

rc.conf

ifconfig_bge0="inet 10.0.0.55"
defaultrouter="10.0.0.150"
ifconfig_bge0_ipv6="inet6 accept_rtadv"
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.1.1.254 netmask 255.255.255.0"
ifconfig_lo1_alias0="inet 10.1.1.1/32"  # srv
ifconfig_lo1_alias1="inet 10.1.1.2/32"  # mail

ifconfig

bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:14:22:bd:3f:05
        inet 10.0.0.55 netmask 0xff000000 broadcast 10.255.255.255
        inet6 fe80::214:22ff:febd:3f05%bge0 prefixlen 64 scopeid 0x1
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
iwi0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 2290
        ether 00:13:ce:56:68:d9
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: IEEE 802.11 Wireless Ethernet autoselect (autoselect)
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.1.1.254 netmask 0xffffff00
        inet 10.1.1.1 netmask 0xffffffff
        inet 10.1.1.2 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33184

The disadvantage with this approach is that it requires a bit of work in /etc/pf.conf:

ext_if="bge0"
tcp_services = "{ ssh, smtp, domain, www, https, pop3, auth, pop3s }"
udp_services = "{ domain }"
JAIL_MAIL="10.1.1.2"
JAIL_SRV="10.1.1.1"
PORT_WWW="{80,443,2222}"
PORT_MAIL="{25,587,465,143,993}"

table <bruteforce> persist

set skip on lo0
scrub in all

!------------------------------[ JAIL ]------------------------------!
nat on $ext_if from lo1:network to any -> $ext_if
rdr pass on $ext_if proto tcp from any to $ext_if port $PORT_WWW -> $JAIL_SRV
rdr pass on $ext_if proto tcp from any to $ext_if port $PORT_MAIL -> $JAIL_MAIL
!------------------------------[ JAIL ]------------------------------!

block all

pass out proto tcp to any port $tcp_services keep state
pass proto udp to any port $udp_services keep state
pass quick proto { tcp, udp } from any to any port ssh \
    flags S/SA keep state \
    (max-src-conn 15, max-src-conn-rate 5/3, \
    overload <bruteforce> flush global)

block quick from <bruteforce>

I am looking for advice as to which method is considered optimal.

Thanks.

 

[dpejesh]

The jail rc script will handle setting up the aliases for you. There's no need to manually do it. An example of how to do it with rc.conf (I haven't got around to doing it with jail.conf yet) is:

jail_enable="YES"
jail_interface="re0"
jail_list="poudriere"

jail_poudriere_rootdir="/usr/local/jail/poudriere"
jail_poudriere_hostname="poudriere"
jail_poudriere_ip="192.168.1.35,lo0|127.0.0.1"
jail_poudriere_mount_enable="YES"
jail_poudriere_devfs_enable="YES"
jail_poudriere_parameters="allow.chflags=1 allow.mount=1 allow.mount.devfs=1 allow.mount.procfs=1 allow.mount.nullfs=1 allow.raw_sockets=1 allow.set_hostname=1 allow.socket_af=1 allow.sysvipc=1 children.max=10 enforce_statfs=1 ip4=inherit ip6=inherit"

This is just an example of how I had poudriere set up (yes its ugly, but it works) in a jail and you can see how the IP is set. jail_interface tells it to use re0 by default, but you can add multiple addresses and bind them to specific interfaces with 'int|ip'. I don't see the benefit of involving pf in this at all, it seems like it's overcomplicating it.

 

[nanotek]

The jail rc script will handle setting up the aliases for you. There's no need to manually do it. An example of how to do it with rc.conf (I haven't got around to doing it with jail.conf yet) is...

I prefer to do things manually, at least at first, before relying on scripts. In any event, thanks for your input. I know your configuration applies to Poudriere, but the principles are the same.

jail_enable="YES"
jail_interface="re0"
jail_list="poudriere"

jail_poudriere_rootdir="/usr/local/jail/poudriere"
jail_poudriere_hostname="poudriere"
jail_poudriere_ip="192.168.1.35,lo0|127.0.0.1"
jail_poudriere_mount_enable="YES"
jail_poudriere_devfs_enable="YES"
jail_poudriere_parameters="allow.chflags=1 allow.mount=1 allow.mount.devfs=1 allow.mount.procfs=1 allow.mount.nullfs=1 allow.raw_sockets=1 allow.set_hostname=1 allow.socket_af=1 allow.sysvipc=1 children.max=10 enforce_statfs=1 ip4=inherit ip6=inherit"

This is just an example of how I had poudriere setup (yes its ugly, but it works) in a jail and you can see how the IP is set. jail_interface tells it to use re0 by default, but you can add multiple addresses and bind them to specific interfaces with 'int|ip'. I don't see the benefit of involving pf in this at all, seems like its over complicating it.

This is the same setup as I had on my previous server, which is described in my original post. That is, jail IP addresses are simply created as aliases on the host external interface (e.g. re0). I don't have a problem with how to create jails or jail IP addresses, but want to know which method network administrators consider "better":

• the standard setup of giving jails alias IPs on the default device;
• cloning a device and giving jails their own network.

I can imagine tighter restrictions could be placed on jails using (b) and I can see the simplicity in using (a). What is considered good practice, though?

 

[dpejesh]

I see what you're getting at now. It sounded like you were manually configuring the alias which isn't necessary, but you know that.

I would think (a) is the most logical way to go. With (a) everything is segregated by IP while with (b) everything is hiding behind the host IP and reverse NAT'd back to the jails. Which I don't see much of a benefit of besides saving IP addresses. And, it adds an extra layer of indirection to the whole setup. With (a) you can move your jails around between hosts without having to involve the network admins at all as long as you're staying on the same network segment (assuming they'd only need to be involved to change reverse NATs or punch new holes in edge firewalls back to your services which were moved to a new IP). I don't see how tighter restrictions can be placed in (b) that couldn't be done with (a). In fact, with (b) you're basically creating a virtual network for your jails that should technically have firewall rules too preventing them from having unrestricted access to each other, which you'd gain from just having them on an external interface and a proper pf configuration.

 

[nanotek]

I typically use (a) too, and think it is the most direct way to the desired result. I know (b) has it's uses, such as in VPS environments with only one IP, but on a local network, assuming addresses aren't limited, (a) is probably the simplest solution. You make a good point about compatibility, relocating setups using (a) would be a little easier; however, there wouldn't be too many extra steps in moving (b) configured jails around either. I suppose that, restricting jails with (b) configuration is somewhat more secure because of the point you make: they are confined to their own network, and if setting up transparent proxies, such as Tor for example, for clients only on that network to use, they might be less susceptible to leaks or contamination than if on a standard (a) setup. Or is that not true?

 

[dpejesh]

there wouldn't be too many extra steps in moving (b) configured jails around either

There wouldn't be, but you asked ' want to know which method network admininstrators consider "better"' so I was trying to look at it from their point of view. With (a) you wouldn't need to involve them at all in most cases.

I suppose that, restricting jails with (b) configuration is somewhat more secure because of the point you make: they are confined to their own network, and if setting up transparent proxies, such as Tor for example, for clients only on that network to use, they might be less susceptible to leaks or contamination than if on a standard (a) setup. Or is that not true?

I don't see how they are less susceptible to leaks. You're actually making them more vulnerable by putting them on basically an unrestricted private network. If one jail is compromised then it will have unfettered access to every other jail unless you set up pf rules between them. If you go that far, then what benefit do you gain over just using those same rules with (a) on the external interface? If you want to properly isolate something like Tor from the rest of your network then use VLANs.

 

[nanotek]

there wouldn't be too many extra steps in moving (b) configured jails around either

There wouldn't be, but you asked ' want to know which method network admininstrators consider "better"' so I was trying to look at it from their point of view. With (a) you wouldn't need to involve them at all in most cases.

Yes, I did; In your view, (a) is considered better practcie than (b) for both security and administrative (even performance) reasons?

I suppose that, restricting jails with (b) configuration is somewhat more secure because of the point you make: they are confined to their own network, and if setting up transparent proxies, such as Tor for example, for clients only on that network to use, they might be less susceptible to leaks or contamination than if on a standard (a) setup. Or is that not true?

I don't see how they are less susceptible to leaks. You're actually making them more vulnerable by putting them on basically an unrestricted private network. If one jail is compromised then it will have unfettered access to every other jail unless you set up pf rules between them. If you go that far, then what benefit do you gain over just using those same rules with (a) on the external interface? If you want to properly isolate something like Tor from the rest of your network then use VLANs.

In the case of leaking data, would (b) not be comparable to VLANs? Further, the network need not be unrestricted and in its default configuration would not be any more or less (un)restricted than (a) in its default configuration. The benefit with (b), I would think, is that by default it is a network separate from the real LAN, which invariably has all types of sensitive information. For all intents and purposes, (b) would be a network absent of such identifying data; a sandboxed network for privacy. Even in the absence of implementing further restrictions with PF for example, if all clients on network (b) abstain from "typical" use, any compromise would only reveal a "fake" network.

If one needs to apply further measures to (b) to gain any advantage on (a) in terms of security (perhaps, privacy/pseudonymity is more apt), you propose running VirtualBox or similar vice any jail setup?

n.b. I'm just questioning, not asserting anything. I appreciate the discussion (and all your posts for that matter), @dpejesh.

 

[dpejesh]

Yes, I did; In your view, (a) is considered better practcie than (b) for both security and administrative (even performance) reasons?

It's a better approach anyway I look at it. The only valid reason I can see for doing (b) is if you're only assigned a single IP on a VPS, or are limited in the amount of IP's you have at your disposal. What happens when you have two jails that both need to run an external service that use the same port, ssh for example? What happens if at 2am you lose that machine completely and need to recreate those jails on other hardware, especially if they don't have a similar setup and need to be split up? How do you handle setting up monitoring which does active probes that require connecting to the jail? What about in a year when you totally forget about it and something breaks and you have to try and remember all the crazy things you did to set it up, or for the poor guy that comes after you when you move onto a new job?

In (b) you also put yourself into a situation where you're relying on other people for you to do your job. If you need to migrate jails from one host to another, in setup (b) you need to possibly make changes to your edge firewalls and DNS because IP's are changing. One of the things I learned early on is if I can eliminate as many outside influences as possible for me to accomplish my job the better off I'd be. A perfect example is I use to work for a big Internet company which had dedicated groups for everything, including DNS. These guys were notorious for spending 75% of their work day playing ping pong and foosball, so when I was spinning up 8 new servers it took them 4 tickets and a week and a half just to get the records created. First ticket they forget one of the hosts, second ticket they misspelled it, third ticket they magically screwed up the host entry above the one they needed to fix, and the fourth ticket they finally got it right. After going through an experience like that you learn not to rely on them for anything, especially if you're in an emergency and need to make changes due to an outage. Not all places are that bad, but if you don't need to involve outside people in your systems work the better off you'll be. Try explaining why it took 2 days for you to get a server back online when it only actually took you an hour to spin up a new jail and recreate it from backups, but then another 2 days were spent trying to force the ping pong champions to work on your P1 ticket to update DNS. Not to mention the time it takes for those changes to propagate. Yes you could do tricks with pf to make it work until they bothered to do their job, but you shouldn't have to in the first place. In an ideal world you would also have proper redundancy, but this is for the sake of argument; or if you're still in a company small enough where you're not fortunate to have the resources required but still have the separation of responsibilities.

In the case of leaking data, would (b) not be comparable to VLANs?

Nope, not the same at all. With VLAN's think of it like having 2 separate ethernet ports on your server which are on separate subnets and you have a jail bound to each of them. The only way they can talk to each other is to go through the network and pass through a router/firewall. With VLANs those cables are merged into one and the separation comes from adding a 'tag' to the packets. 802.1Q. Even though you have 2 jails running on the same host, they're technically not on the same network. In setup (b), all the traffic leaving the host is coming from the host's IP and its all hitting the same network.

The benefit with (b), I would think, is that by default it is a network separate from the real LAN, which invariably has all types of sensitive information. For all intents and purposes, (b) would be a network absent of such identifying data; a sandboxed network for privacy. Even in the absence of implementing further restrictions with PF for example, if all clients on network (b) abstain from "typical" use, any compromise would only reveal a "fake" network.

It's not really separate from the LAN though. All traffic they send out hits the network anyway, regardless of if its NAT'd to the hosts IP or a dedicated IP. Sure you can setup pf rules to block their outgoing traffic, but you can do the same thing in (a). I'm not seeing how privacy fits in to this. If you're afraid of someone being able to run tcpdump within the jail and see all network traffic to the host (including all other jail traffic), that's where devfs comes into play. You tell your jails to use devfs_ruleset=4 which prevents the required device nodes from being created in /dev. I've never tried this, but I'm willing to bet in setup (b) the jails would still be able to tcpdump the external interfaces network traffic if you don't setup the proper devfs rules.

If one needs to apply further measures to (b) to gain any advantage on (a) in terms of security (perhaps, privacy/pseudonymity is more apt), you propose running VirtualBox or similar vice any jail setup?

In general, any system that uses VT should be more secure, but at what cost? Jails are extremely lightweight and easy to manage and integrate into configuration management while VirtualBox requires more resources and can be much more of a headache to work with. It really just depends on what you need as to which is more suitable, but ones not inherently better than the other.

n.b. I'm just questioning, not asserting anything. I appreciate the discussion (and all your posts for that matter), @dpejesh.

I didn't read your response as being assertive. I know I can come off a bit dickish, but that's just my inability to write properly.

 

[kpa]

I didn't read your response as being assertive. I know I can come off a bit dickish, but that's just my inability to write properly.

It's also known as inability to sugar coat your message for general consumption §e

 

[throAU]

2c.

KISS

Keep it simple, stupid.

If you have IP address space available, I'd use option A, as to me it seems cleaner, less complex network configuration wise (thus, less prone to error) and more "self contained".