Solved Jail NAT vs Bridge

[Farhan Khan]

Hi all,

I want to create a jail with a publicly routable IPv6 address and a 192.168.0.0/16 IPv4 address. I also do not want the have the interface shared with the base OS, such that listening on port 80 on the base also listens to port 80 on the jails.

I do not see where that is in any documentation.

Separate topic: Also the Jail and Ezjail handbook sections appears to be out of date. They still uses /etc/rc.conf.

 

[wblock@]

The first section of the Jails chapter is out of date. It was suggested at BSDCan that all of the Jails chapter except the ezjail section be ripped out. ezjail itself does still use /etc/rc.conf, so that section is up to date with the software. When or how that might change has not been stated.

As for the original question, the base system is in charge of the jails. If you are concerned that the host can possibly eavesdrop on the jails, then a jail might be the wrong thing to use. Other types of virtualization offer more separation, but the host will always have the highest access. Running any type of virtualization on a host that is not trusted is kind of wishful thinking. It must be assumed that whoever set up the host was able to do anything they wanted, including modifying the jail system or installing special hardware that provides unwanted access to the VMs.

 

[Farhan Khan]

That's for acknowledging that it is out of date. As a pseudo-new user, it was driving me insane!
In the interim, do you recommend any other documentation?

I am not concerned that the host system could eavesdrop, I am concerned that the bases' services bind on the same IP as the child jail.

So, if the base OS's is 192.0.2.1 and a jail has the IP 192.0.2.2, the base will have a shared interface with the jail with the same IP. If the base OS opens a port on 0.0.0.0:4444, it will also bind on the Jail's 192.0.2.2:4444. The Jail cannot also bind on port 4444 if the Base does the port is already taken.

That seems highly problematic.

 

[robroy]

Farhan, I too found the network stack sharing that goes on between a host and its jails unnerving at first. Yet having toyed around with jails for a few years, I now feel that this sharing represents its own kind of simplicity, relative to other virtualization options--it's very light-weight. And the Unix service configuration and networking know-how that's required to be comfortable with it's widely applicable to other kinds of non-jail situations. The same isn't necessarily true for the knowledge required to use other virtualization options well.

Once jails are being used, the conventional things to run as few services as possible directly on the host. This makes the port number conflict scenario a lot less problematic than it seems at first.

For the few services that make sense to keep running on the host, like sshd, configuring them to listen only on the host's IP address (for instance, 192.168.16.1:22 instead of *.22) is an easy, one-time step when the host's first installed. And as you probably know, this kind of service configuration's not necessary within the jails; the jails are only able to use the IPs they're configured to use.

I believe FreeBSD 11 will include a stable VIMAGE feature, which will allow each jail to have its own virtual network stack. This ought to obviate the problem you're concerned about, though at what cost (in complexity) I'm not sure; I haven't tried it.

 

[Farhan Khan]

Robroy (not sure how to do a link to your name), I suppose the manual method also works. FreeNAS creates a unique bridged interface for each jail, which does not have an IP address.

Then final relevant question - How do I give my Jail network connectivity? I assigned it an IP address, but the routing rule is as follows:

# ifconfig
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether f2:3c:91:ae:7a:0f
    inet 192.168.0.11 netmask 0xffffff00 broadcast 192.168.0.255
    inet6 2600:3c03:e000:12c::d prefixlen 64
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

My routing rules do not let me get online, as you can see:

# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
192.168.0.11       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire

What's the proper way to setup routing?

 

[robroy]

Robroy (not sure how to do a link to your name)

The Profile Linking section on this BB Codes reference page shows an example.

I suppose the manual method also works. FreeNAS creates a unique bridged interface for each jail, which does not have an IP address.

That's interesting; thanks. I didn't know that.

Then final relevant question - How do I give my Jail network connectivity? I assigned it an IP address, but the routing rule is as follows:

Network traffic that's associated with a jail's IP(s) is actually subject to the same routing table that the host uses; there's only one routing table. When I run netstat -rn from within my own jails (which have normal network access), I see the same peculiar output that you're seeing. I think that's normal behavior for netstat when jailed, strange as it seems.

What's the proper way to setup routing?

Routing should be configured only on the host; it's neither necessary nor possible to do routing configuration from within jails.

With regard to why networking from within your jail's not working as desired, I'd be happy to eyeball your configuration if you felt like posting some command outputs. I lack IPv6 knowledge, but maybe I'd luck out with the rest and manage to be helpful.

I'd find these command outputs (run on the host, and not from within the jail) of interest:

• cat /etc/rc.conf
• cat /etc/jail.conf
• ifconfig -a
• netstat -rn
• jls

Happiness to you, Fahran.

 

[Farhan Khan]

The issue was that I failed to set a DNS server in /etc/resolv.conf. Worked like a charm as soon as I did that.
Yes, it does seem odd, but it works. And that's good enough for me. Thank you!

 

[robroy]

'glad to hear it, and you're welcome.