Explicitly assign a loopback interface in a jail

[Tsuroerusu]

When reading the release notes for 11.1-RELEASE, I noticed this new change:

The jail(8) utility has been updated to allow explicitly-assigned IPv4 and IPv6 addresses to be used within a jail. [r316944] (Sponsored by Multiplay)

I like to dedicate entire NICs to a single jail by using multiple routing tables (fibs), and in case anybody else is doing that and would like dedicated loopback interfaces in their jails (Given the change to the jail sub-system in 11.1-RELEASE), this is how I made it work.

My motivation for doing this was to avoid having php-fpm exposed on my jail's network-facing IP-address. When installed via pkg(), the daemon is configured to use TCP by default and to listen on 127.0.0.1. As I see/understand it, this presents a potential security hazard if a jail has only a single LAN-facing IP-address assigned as all activity to the loopback address would be re-mapped to that LAN-facing IP-address by the kernel, and thus exposing the daemons listening to 127.0.0.1 on the network. This, of course, is less on an issue if running pf(), but in my gut its still a potential problem. I could imagine this being a similar case with MariaDB etc.

While tinkering with this, I found that I was getting errors like the following (I used the jail's sshd as a test):

# ssh 127.0.1.1
ssh: connect to host 127.0.1.1 port 22: Can't assign requested address

# ssh localhost
socket: Protocol not supported
ssh: connect to host localhost port 22: Protocol not supported

One thing that tripped me up was, as I discovered, that the loopback interfaces for the jails must be configured to use the same fib as the jail in general and its network-facing interface/IP-address.

In this example, I am adding dedicated loopback devices for two jails:

In rc.conf, add the following:

cloned_interfaces="lo1 lo2"
static_routes="jail1_lo jail2_lo"
route_jail1_lo="-net 127.0.1.0/8 -iface lo1 -fib 1"
route_jail2_lo="-net 127.0.2.0/8 -iface lo2 -fib 2"

The full setup looks something like this:

cloned_interfaces="lo1 lo2"
static_routes="jail1_lo jail2_lo dmz1_if dmz1_gw dmz2_if dmz2_gw"

route_jail1_lo="-net 127.0.1.0/8 -iface lo1 -fib 1"
route_jail2_lo="-net 127.0.2.0/8 -iface lo2 -fib 2"

route_dmz1_if="-net 10.0.2.0/24 -iface igb1 -fib 1"
route_dmz1_gw="default 10.0.2.1 -fib 1"
route_dmz2_if="-net 10.0.3.0/24 -iface igb2 -fib 2"
route_dmz2_gw="default 10.0.3.1 -fib 2"

After defining these static routes for the fibs, you need to adjust jail.conf() to assign IP-addresses to the jail on both the loopback interfaces and the NICs. For each jail, replace your existing ipv4.addr line with the following:

ip4.addr = "lo1|127.0.1.1/8", "igb1|10.0.2.5/24";

For me, jail1 would look like the following:

jail1 {
        path = "/jails/jail1";
        mount.fstab = "/etc/fstab.jail1";
        host.hostname = "jail1.myserver.dk";
        exec.fib = "1";
        ip4.addr = "lo1|127.0.1.1/8", "igb1|10.0.2.5/24";
}

Also, if you're using pf() make sure to skip filtering on all the loopback devices:

set skip on lo0
set skip on lo1
set skip on lo2

Finally, edit the jail's hosts file to reflect that it now has its own dedicated loopback address:

127.0.1.1    localhost localhost.mydomain

You may need to reboot to apply these changes properly.

Having spent hours working this out, I hope this was useful to somebody out there.

 

[Mikalai]

Do you filter ports and direction of connection between jails?

I am able to recreate scenarios with both no inter-jails communication, and unrestricted inter-jails communication. But I cannot fine filter in between.

Can you show netstat -r with all ips that are related to jails. Chances are that igb's addresses are connected to lo0, which bypasses any filtering due to set skip on lo0.

 

[amartin]

I would also like to configure the same behavior, binding daemons to localhost so as to not expose them over the public interface. I am using iocage with Shared IP mode and I believe I have achieved the desired result:

1. configure both the public interface (e.g vtnet0) and localhost interface (lo0) in the jail; note that the public interface MUST come first:

# iocage set ip4_addr="vtnet0|192.168.1.111/24,lo0|127.0.1.1/8" myjail

2. start the jail

# iocage start myjail
# iocage console myjail

3. Set the "localhost" entry in /etc/hosts in the jail to use the IP you assigned to lo0, 127.0.1.1 in this case

4. Configure your public service (in this case nginx, listening on port 80) to listen on all interfaces or the vtnet0 interface and your private service (in this case listening on port 8080) to listen on "localhost" or 127.0.1.1

5. restart the jail to make sure the changes to /etc/hosts are applied:

# iocage restart -s myjail

6. confirm that the public service is listening on *:80 or the public interface and the private service is listening on 127.0.1.1:8080:

# iocage exec myjail netstat -an
netstat: kvm not available: /dev/mem: No such file or directory
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 *.80                   *.*                    LISTEN
tcp4       0      0 127.0.1.1.8080         *.*                    LISTEN
tcp4       0      0 *.22                   *.*                    LISTEN

7. run nmap on the host to see if the desired ports are open (80) and closed (8080) on the public IP:

# nmap -PN -p8080 192.168.1.111

Starting Nmap 7.40 ( https://nmap.org ) at 2017-08-31 22:29 CDT
Nmap scan report for 192.168.1.111
Host is up (0.000074s latency).
PORT     STATE  SERVICE
8080/tcp closed http-proxy

Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds
# nmap -PN -p80 192.168.1.111   

Starting Nmap 7.40 ( https://nmap.org ) at 2017-08-31 22:29 CDT
Nmap scan report for 192.168.1.111
Host is up (0.000075s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds
# nmap -PN -p80 127.0.1.1

Starting Nmap 7.40 ( https://nmap.org ) at 2017-08-31 22:29 CDT
Nmap scan report for 127.0.1.1
Host is up (0.000076s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds
# nmap -PN -p8080 127.0.1.1 

Starting Nmap 7.40 ( https://nmap.org ) at 2017-08-31 22:29 CDT
Nmap scan report for 127.0.1.1
Host is up (0.000088s latency).
PORT     STATE SERVICE
8080/tcp open  http-proxy

I believe this has the desired effect of protecting the private service listening on 8080 from the public interface, correct? Are there any caveats with this setup? The only thing I can think of is that other jails on the same host could access the port if they scanned the 127.0.0.1/8 subnet. You can use ipfw to block traffic from other hosts on 127.0.0.0/8 as follows:

#!/bin/sh
ipfw -q flush

# allow previously-established connections
ipfw add 10 check-state
ipfw add 20 allow tcp from any to any established

# allow traffic to/from public subnet
ipfw add 150 allow all from any to 192.168.1.0/24 in
ipfw add 160 allow all from 192.168.1.0/24 to any out

# repeat these 3 lines for EACH jail lo0 IP you configure:
# allow all traffic to jail's private IP except from other IPs on 127.0.0.0/8
ipfw add allow tcp from 127.0.1.1 to 127.0.1.1 in
ipfw add allow tcp from not 127.0.0.0/8 to 127.0.1.1 in
ipfw add allow tcp from 127.0.1.1 to any out

It would be nice if there were a more elegant way to block traffic between IPs on 127.0.0.0/8, but at least having these 3 rules per jail does work.