Networking and jails

quamenzullo

quamenzullo

Hello,

I still have some questions about networking and jails. I could not find the answers in the documentation or forums. I hope my questions are not too dumb.

1. The networking inside jails seems to partly rely on the networking of the host. To get "more" networking features, it is necessary to use things like vnet, vimage... Am I right?

2. The documentation and several posts on the forum advise to bind the jails' IP addresses to lo1, what is also a loopback interface. Why does the jails' IP addresses have to be bound to a loopback interface? Does it matter at all? Is there no way to use another logical interface what wouldn't be a loopback one?

3. Several posts on the forums advise to give each jail an address in 127/8. These addresses are reserved for loopback. What is the purpose behind such a choice (instead of addresses in 192.168/16 or in 10/8)?

4. The documentation of ezjail says "To keep jail loopback traffic off the host's loopback network interface lo0, a second loopback interface is created by...". OK, but why is it necessary to keep the jail loopback traffic off the host's loopback network? (Security?)

5. With pf configured the right way on the host, the jails do reach internet (e.g. whois works and nginx answers well to requests), but I do not understand how. From inside a jail, ifconfig outputs:

Code: 
root@web0:~ # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether **:**:**:**:**:**
  media: Ethernet autoselect (100baseTX <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet 10.1.1.64 netmask 0xffffffff
root@web0:~ #

So, no IP address is bound to em0, though the jail can "see" it. The jail's own IP address (10.1.1.64) is bound to lo1 (what is a loopback interface). Moreover, if I delete the defaultrouter setting from the jail's /etc/rc.conf, the jail can still reach internet.

So, how do the jails reach internet?

6. Someone mentionned it is possible to setup the host and jails in order to let the jails accessing internet without using NAT (thanks to pf). But I couldn't find how it is possible to do so. (I will use pf anyway, so this question is only caused by curiosity).

Thanks for any hints!
 
Jails do not rely "partly" on the host for their networking but absolutely and completely, jails have no network stack of their own so the defaultrouter etc settings have no effect inside the jails. The only thing that is different inside a jail is that only a limited set of IP addresses are selectable by a process running in the jail.
 
quamenzullo said:
4. The documentation of ezjail says "To keep jail loopback traffic off the host's loopback network interface lo0, a second loopback interface is created by...". OK, but why is it necessary to keep the jail loopback traffic off the host's loopback network? (Security?)
Click to expand...
When writing that section, the cluster admins advised me that separating jail loopback traffic from the host made things greatly better. From a security point of view, jails should not be sharing the host loopback, either.
 
wblock@ said:
When writing that section, the cluster admins advised me that separating jail loopback traffic from the host made things greatly better. From a security point of view, jails should not be sharing the host loopback, either.
Click to expand...

Loopback lo0 is often unfiltered because writing firewall rules for it that actually work is quite tedious and difficult, the same interface and IP address work as both the source and destination of traffic.
 
  • Thanks
Reactions: Oko
kpa said:
Jails do not rely "partly" on the host for their networking but absolutely and completely, jails have no network stack of their own so the defaultrouter etc settings have no effect inside the jails. The only thing that is different inside a jail is that only a limited set of IP addresses are selectable by a process running in the jail.
Click to expand...

OK, so I guess this answers also to question 5.

I guess the answer to question 6. is: it's necessary to do NAT in order to let the jails access to internet (but it is possible to do NAT without pf).

And about these loopback questions: I should just remember that it's just like that for deep technical reasons... (anyway, it works, my questions were only motivated by curiosity).
 
quamenzullo said:
I guess the answer to question 6. is: it's necessary to do NAT in order to let the jails access to internet (but it is possible to do NAT without pf).
Click to expand...
This is possible. During my first tests with jails I have had the line in /usr/local/etc/ezjail/www as
Code: 
export jail_www_ip="lo1|10.0.0.1,bge0|192.168.0.1"
The system has just one network adapter and with this config it has shown up with the additional address 192.168.0.1. I have no idea if this is a good idea or completely stupid. It would be good to hear the opinion form more experienced persons than myself. This would require a firewall in the jail if required and so on. But it safes NAT. Now I have changed the config to
Code: 
export jail_www_ip="lo1|10.0.0.1"
only and do NAT on the host.
 
If you can get connectivity using "normal" routing NAT would not be needed. The reason most people use NAT is simply because they only have one outside address and no way to route the 'internal' IP addresses.
 
There are so many streets leading to Rome. In the days before I tried the how-to-do of hukadan about jailing www/firefox and of kpa about ports-mgmt/poudriere I have had local-unbound from the base system running and additionally www/privoxy to filter http traffic. Currently I have a jail for firefox. The packages come from a separate set in ports-mgmt/poudriere. This makes my main system lean. Rebuilding by main system by devel/poudriere is much faster. The firefox jail is updated only if one of its ports is vulnerable. A second jail holds basically texlive-full from packages and not from ports. But I wanted to have the simple way of handling dns and http back, and without NAT if possible.

Changing from local-unbound to dns/unbound allow the jails to use the cache via lo1 interface. So far so good. For http it is possible to configure www/privoxy to listen on lo1. So www/privoxy could be on moved from the www jail the main system. This enables the option to use packages for browsers in different jails using just one filtering proxy, too. The remaining question how to deal with pkg could be resolved as well because pkg can be configured to use http using a proxy. This proxy is of course www/privoxy on the main system.

This makes for my current system NAT unnecessary because the traffic from and to the jails in the current setup can be handled by the proxies on the main system. On top of that a common firewall is possible without the need to think about NAT.
 
supportsobaka said:
Could you please share how you forwarded 10.1.1.64 from lo1 to the real ip of em0?
Do you know rules for ipfw?
Click to expand...
Please keep in mind that you're responding to a thread which is roughly 2 years old. So there's no guarantee that the user you responded to is actually still active.

Alas: the best way to do this would be NAT and the best source of information on this (IMO anyway) is the FreeBSD handbook. Pay special attention to Chapter 29; this one covers all supported firewalls.

Next to that also keep in mind that manual pages on FreeBSD are actually what they claim: reliable source of information which is usually found in a manual. Ergo ipfw(4) (so using the man ipfw command) can also provide some good hints with this.
 
I have checked all these and tons of others sources. Unfortunately, handbook does not provide up-to-date information. There is old natd method described and nothing useful about kernel nat. Of course I read man ipfw, but it was hard to figure out the correct setup. It is virtually impossible to find information how to make jail with shared IP on lo1 interface go to the real IP on external interface. I will try VIMAGE/vnet, probably it will be easier task.
 
To enable NAT support at boot time, the following must be in /etc/rc.conf:
Code: 
gateway_enable="YES"# enables the gateway
natd_enable="YES"# enables NAT
natd_interface="rl0"# specify interface name of NIC attached to Internet
natd_flags="-dynamic -m"# -m = preserve port numbers; additional options are listed in natd(8)
Click to expand...
And:
$cmd 005 allow all from any to any via xl0 # exclude LAN traffic
$cmd 010 allow all from any to any via lo0 # exclude loopback traffic
$cmd 100 divert natd ip from any to any in via $pif # NAT any inbound packets
# Allow the packet through if it has an existing entry in the dynamic rules table
$cmd 101 check-state
Click to expand...
What do you mean "impossible"? This makes it pretty obvious that you'll need a divert rule to set up NAT. Together with the natd daemon. I assume that the only reason natd is being used is due to IPFW. However, when I check ipfw(8) I also see mentioning of the nat keyword. Here is a quote directly from the manualpage:

First redirect allthe traffic to nat instance 123:

ipfw add nat123 allfrom any to any

<CUT>
To see configuration of nat instance 123:

ipfw nat 123show config

Toshow logs of all the instances in range111-999:

ipfw nat 111-999 show

Tosee configurations of all instances:

ipfw nat show config
Click to expand...
And if you don't like using that you can always switch to another firewall. For example, my personal favorite PF only needs the enabling of forwarding and a few proper NAT rules, no natd required.

Sorry to say but I can't help get the feeling that you didn't really give this much effort.
 
ShelLuser said:
natd daemon
Click to expand...

I told about kernel nat (that one that you quoted from manpage). I don't want natd daemon with its overhead, it's 2018 here.

ShelLuser said:
Sorry to say but I can't help get the feeling that you didn't really give this much effort.
Click to expand...

Seems you know plenty sources where explained how to get non-vnet jail with shared local IP on lo1 (say 192.168.0.5) access Internet, when physical interface (say, em0) is allowed to have only one public IP (say, 8.8.8.8) and default router is, say, 8.8.8.1 without NATing all traffic that comes to machine? With ipfw and kernel nat. No natd, no PF.
 
As you asked for pf configuration in your first post, here are the relevant lines to allow internet access to the jails (but I am no expert and cannot guarantee this would be the correct configuration in other cases (nor that this was the best in my case)):

Code: 
# INTERFACES
ext_if = "em0"
int_if = "lo1"
jailnet = $int_if:network

# NAT: jails access to internet
nat on $ext_if from $jailnet to any -> ($ext_if)

And I will mention that I have been well advised to read the Book of PF (by Peter N. M. Hansteen).

Now, it looks like your requirements:
With ipfw and kernel nat. No natd, no PF.
Click to expand...
and actual questions are not exactly related to the original topic. Wouldn't this be better to move these new questions to a separate topic?
 
I find a lot of the IPFW examples quite complex. I think they are taking advantage of its stateful functionality which I don't quite understand enough. For my Jails I use the following:

Code: 
ipfw nat 200 config if bge0 reset same_ports
ipfw add 50 nat 200 all from any to any

I believe the first line enables NAT for the bge0 interface and the second line tells all traffic to go through that NAT rule.

If you want to forward incoming ports to the jails (i.e 2000-3000) then something like this instead:

Code: 
ipfw nat 200 config if bge0 reset same_ports redirect_port tcp 127.0.1.2:2000-3000 2000-3000
ipfw add 50 nat 200 all from any to any

Note the 127.0.1.2 is the address of my loopback alias (lo1) which the Jails are bound to.
 
kpedersen said:
ipfw add 50 nat 200 all from any to any
Click to expand...
This will nat *all* traffic and ruin all other rules.
ipfw nat is really taught! Handbook doesn't give any examples at all.
All other examples about ipfw nat are pretty similar to your - they assume that you run a gateway where you just need to nat all traffic. Friends who uses FreeBSD many years as me couldn't figure out rules too.
After 4 days of my "didn't really give this much effort" I think I have to forget about ipfw nat and better to run jail on public ip6 that DC provides with generosity.
 
You must log in or register to reply here.
Back
Top