Solved Setting up VLAN for Jails

K

kjpetrie

I have a virtual server with a single IP address on which I wish to set up jails using sysutils/ezjail. At present I'm testing the configuration on a VirtualBox VM install of FreeBSD.

The sysutils/ezjail page in the Handbook suggests setting up the jail with an alias to the external NIC (in my case em0) in the same subnet as the allocated IP, but that is visible from outside and would doubtless upset my hosting company if I started pinching IPs that aren't mine! The other option is to use the lo1 interface with NAT and redirection, but I understand that could expose local processes which should stay private.

I therefore decided to try a virtual bridge with NAT and redirection, but I'm not sure how that works in FreeBSD.

1. Do I need taps or can I use aliases directly on the bridge?
2. On the assumption I might need taps I have created some, though if I assign them IPs they remain with status "no carrier". What do I need to do to "connect" the tap?
3. If I assign an alias directly to the bridge I still can't forward it to my external NIC.

Here are the relevant parts of /etc/rc.conf:
Code: 
ifconfig_em0="inet 10.0.2.141 netmask 255.255.255.0"
cloned_interfaces="${cloned_interfaces} lo1 bridge0 tap0 tap1 tap2 tap3 tap4"
ifconfig_bridge0="ether 10:00:00:00:00:06 addm tap0 addm tap1 addm tap2 addm tap3 addm tap4"
ifconfig_lo1_aliases="inet 127.0.1.1-5/32"
ifconfig_bridge0_aliases="inet 10.20.0.0-5/24"
,

/etc/pf.conf
Code: 
ext_if = "em0"
int_if = "bridge0"
jails_out = "{ domain, auth, smtp, http, https, cvspserver, 5999, 8000, 8080 }"
localnet = $int_if:network
WEB_IN_ADDR="10.20.0.1"

nat on $ext_if from $localnet to any -> ($ext_if)

rdr pass on $ext_if inet proto tcp to port http -> $WEB_IN_ADDR port http
rdr pass on $ext_if inet proto tcp to port https -> $WEB_IN_ADDR port https
,

/usr/local/etc/ezjail/sharedserver:
Code: 
export jail_sharedserver_ip="lo1|127.0.1.1,bridge0|10.20.0.1"

and, finally, ifconfig:
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
  ether 08:00:27:4f:e9:cb
  inet 10.0.2.141 netmask 0xffffff00 broadcast 10.0.2.255
  inet6 fe80::a00:27ff:fe4f:e9cb%em0 prefixlen 64 scopeid 0x1
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet 127.0.1.2 netmask 0xffffffff
  inet 127.0.1.3 netmask 0xffffffff
  inet 127.0.1.4 netmask 0xffffffff
  inet 127.0.1.5 netmask 0xffffffff
  inet 127.0.1.1 netmask 0xffffffff
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  ether 10:00:00:00:00:06
  inet 10.20.0.0 netmask 0xffffff00 broadcast 10.20.0.255
  inet 10.20.0.1 netmask 0xffffffff broadcast 10.20.0.1
  nd6 options=9<PERFORMNUD,IFDISABLED>
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 11 priority 128 path cost 2000000
  member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 10 priority 128 path cost 2000000
  member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 9 priority 128 path cost 2000000
  member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 8 priority 128 path cost 2000000
  member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 7 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:d0:05:00:00
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier
tap1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:d7:05:00:01
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier
tap2: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:dd:05:00:02
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier
tap3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:e3:05:00:03
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier
tap4: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:ea:05:00:04
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier

Any pointers welcome. Thanks.
 
A bridge is a layer 2 connection. By bridging the interfaces you might as well just bind the jail to the external interface, it's going to have the same effect.
 
In Linux I can set up a floating bridge to act as a network between virtual machines, and then use packet forwarding/NAT to connect it to the external interface. Can't I do something similar in FreeBSD?

I want something to connect the jails together, which can then be forwarded to the single external interface without the Jails' IPs being visible on the Internet.

I would have expected both kernels to have similar routing capabilities, even if the terminology is a little different.
 
Why do you need the bridge? The kernel can route traffic between interfaces just fine as long as you have turned on forwading in rc.conf(5). After that you can use your favorite firewall (PF or IPFW in FreeBSD) to hide those jail IP addresses with outbound NAT. For example with PF the NAT is usually a single line operation:

Code: 
nat on $ext_if inet from !$ext_if to any -> $ext_if
 
Because I have only one interface (not counting loopback, which is apparently subject to security concerns) and I need to create a second (virtual) one ("!$ext_if" in your sample line) to enable routing between them.
 
kjpetrie said:
Because I have only one interface (not counting loopback, which is apparently subject to security concerns) and I need to create a second (virtual) one (! $ext_if in your sample line) to enable routing between them.
Click to expand...

You shouldn't need a bridge to enable connectivity between the interfaces and it's actually harder to write firewall filter rules for the bridge if you want to separate jails from each other and from the external interface. Where did you get the idea that a single external interface is a "security concern"?
 
I've read so many how-tos and posts over the last two days that it's difficult to locate the exact one which said lo1 should not be forwarded to the outside world because that would provide access to system daemons running inside the jail which should not be exposed to the Internet. However, on that basis I decided I shouldn't use lo1 and should instead create another interface for the jail(s) to use.

I can't connect the jails directly to the external interface because on the live server that would mean choosing aliases in the same subnet as my server's IP address (which is a public one in a public subnet). These aliases would be visible outside and therefore would clash with the IPs of other customers of the hosting company. I therefore need a separate internal interface for the jails so I can give them private IPs.

I therefore would like to know how to create that internal interface to connect the jails, which could then be forwarded/NATted by PF.

Alternatively, if there's another safe way to do it I'm open to suggestions.
 
While I was typing that last post I got an e-mail containing:
"Just so that were are on the same page, you're not talking about Layer 2 connectivity between the jails? If you are then why do you need that?"

That sent me to Wikipedia to look up the layers. I think the connectivity I would need between jails needs to be no greater than between jails and the outside world - probably level 3 would do.

At present I have a single server running name-based virtual hosting for different customers. I would like to do two things:

1. be able to separate some of the websites into their own jails,
2. Create a website for a political campaign and split its functions between jails to make it more robust against attack by motivated opponents.

All this from a single public IP address!
 
You just want to do NAT. So use the loopback interface for the jails and then pf to forward the neccessary ports to the jails.

For example:
jail.conf
Code: 
root@dolw17srv01:~# cat /etc/jail.conf
##########################################
###### General Settings  ######

# Jailstart
exec.start="/bin/sh /etc/rc";

# Jailstop
exec.stop="/bin/sh /etc/rc.shutdown";

# Clean
exec.clean;

# Devfs 
mount.devfs;

# Network interface
interface="lo0";

##########################################
###### Per Jail Settings  ######
dns {
  host.hostname = "dns.home";
  path = /jails_new/dns;
  ip4.addr = "10.0.0.1";
}

pf.conf
Code: 
#INTERFACES
ext_if="re0"
jails = " { 10.0.0.1 } "

# Name and IP of jails
Jail1="10.0.0.1"

# NAT to the outside
nat log(all, to pflog0) on $ext_if from $jails to any -> ($ext_if)

# REDIRECT
rdr pass on $ext_if inet proto tcp to port 53 -> $Jail1 port 53
rdr pass on $ext_if inet proto udp to port 53 -> $Jail1 port 53

Regards
Markus
 
I still not understand, what you mean with "security implications".

You could adapt the packet filter to filter each packet on the loopback-interface, so that the jails can not communicate with the outside at all or between each others.

Regards
Markus
 
I have no further information, I'm afraid. I cannot find the document in which I read the 'advice', which seemed plausible at the time. It was something to do with system daemons listening on loopback which shouldn't be exposed to the Internet through packet forwarding, but I don't know whether they meant on the host or in the jails, or which loopback device.

Given the widespread use of loopback for this purpose in practice and the lack of any advice on this forum not to do so, I suspect whoever wrote that was misinformed and spreading misinformation. Let's face it. There's a lot of that on the Internet too.
 
They were probably talking about lo0 i.e. 127.0.0.1. Daemons bound to that interface and address should never be forwarded. That would indeed cause security issues. But jails are bound to a cloned interface using a different IP range. Because daemons that bind to lo0 are on a different interface (that's not forwarded) there's no issue.
 
You must log in or register to reply here.
Back
Top