Jail with unique IPs and VNET

Y

yzbythesea

Hi

My goal is to setup VNET Jails with their own IP in my home network. For example, my home router has IP 192.168.1.1 and this FreeBSD server has IP 192.168.2.1
I would like my Jail to have unique IP in the home LAN, e.g: 192.168.2.2, 192.168.2.3 and etc.

The configs I am having right now:

On the host:​

/etc/rc.conf
Code: 
defaultrouter="192.168.1.1"
gateway_enable="YES" # do I need this enabled?
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.2.1/24 addm em0 up" # do I need to add alc0 into bridge0?
ifconfig_em0="up" # can I also assign IP to this em0, which is my actually physical interface

/etc/jail.conf.d/example.conf
Code: 
example {
    exec.prestart = "ifconfig epair0 create up";
    exec.prestart += "ifconfig bridge0 addm epair0a up";
    exec.poststop = "ifconfig epair0a destroy";
    vnet;
    vnet.interface = "epair0b";

    .......... # other configs
}

Inside my example Jail:​

/etc/rc.conf
Code: 
defaultrouter="192.168.2.1"
ifconfig_epair0b="192.168.2.2"

I have couple questions

  1. Should I add em0 (the physical interface of my server) to the bridge if I enable gateway service? I thought gateway service can auto-forward ethernet packets from physical interface to the bridge.
  2. Do I need to assign an IP to em0?
  3. Do I need to assign an IP to epair0a?
  4. If I have multiple VNET Jails, bridge is mandatory or is there a way I can achieve without bridge for having VNET Jails with unique IPs?
 
I normally have it setup this way:
/etc/rc.conf
Code: 
ifconfig_em0="inet 192.168.1.2/24"
defaultrouter="192.168.1.1"
cloned_interfaces="bridge0 epair0"
ifconfig_bridge0="inet 192.168.100.1/24 addm epair0a up"
ifconfig_epair0a="up"
gateway_enable="YES"

Then each /etc/jail.conf.d/*.conf will have an IP address in the 192.168.100.0/24 range and default gateway set to 192.168.100.1

For /etc/pf.conf
Code: 
ext_if="em0"
br_if="bridge0"
nat on $ext_if inet from 192.168.100.0/24 to any -> $(ext_if:0)
pass quick on $br_if
 
I've always found this article quite helpful. I've used it to run a web server in a jail, using PF to NAT traffic in and out of the jail without problems. It's much like mumu's description with the addition of using PF to NAT.

The author suggests ezjail, but I've always done it by just making a jail as per the handbook's instructions and going from there. It's just cloning an interface, putting the ip in /etc/jail.conf, and setting up pf.
 
scottro Looks like this approach will not allow your jail to have its own IP in my home LAN.
So if my homeserver has IP: 192.168.2.1. And my Jail has internal IP: 192.168.2.100, but this 192.168.2.100 cannot be called by other PCs on my home network. I can instead forward request by port, e.g. if I NAT 8080 port from 192.168.2.100 to 192.168.2.1, then other PCs making a call to 192.168.2.1:8080 can reach my Jail. Is that correct?
 
I'm sorry, I missed that part. I read have its own IP and skimmed the rest. That's true, the IP wouldn't be on the LAN, it would be a separate subnet. To answer your question about what does the NAT rule do
if your LAN is 192.168.1.0, and the cloned interface network is 192.168.100.0, it would send packets out over your hosts network card that goes to the internet. So you can do pkg install from inside the cloned interface network and it would go out
through your host's network NIC and get any required package.
 
yzbythesea said:
For example, my home router has IP 192.168.1.1 and this FreeBSD server has IP 192.168.2.1
Click to expand...
These are not on the same network, unless you have set up a /16 instead of the more common /24. They might be physically connected to the same "network", but that's generally a bad idea (can be done, but each subnet should ideally be separated with VLANs). Your rc.conf shows a /24, so these are two separate networks and you need to route between them.

yzbythesea said:
So if my homeserver has IP: 192.168.2.1. And my Jail has internal IP: 192.168.2.100, but this 192.168.2.100 cannot be called by other PCs on my home network.
Click to expand...
Neither can 192.168.2.1 be reached from any other host on 192.168.1.0/24 unless you have set up a bunch of static routes (or have set up your router to route between those two subnets).
 
SirDice said:
These are not on the same network, unless you have set up a /16 instead of the more common /24
Click to expand...
Ah I meant to have them in the same home network. I will modify my FreeBSD server to have IP of 192.168.1.2 instead. So my home router is 192.168.1.1 and my server is 192.168.1.2.

My goal is to have a separate network stack for my Jail so I can apply pf rules separately and also monitor connections. But I can still reach to my Jail from other PCs on my home network.

I think one proposal as mumu mentioned is,
For example, I host a service in the Jail listening on :8080 and port forward to my FreeBSD server so any call to 8080 port will be redirected to my Jail. Then I can call my Jail service from my other PC, by 192.168.1.2:8080

The original idea I have is to make Jail have a IP available in home network, e.g. assign 192.168.1.10 to my Jail and I call the service by 192.168.1.10:8080, but this is not recommended. Is that correct? SirDice
 
yzbythesea said:
The original idea I have is to make Jail have a IP available in home network
Click to expand...
Yes, that's how you should do it. Bind the bridge to the same interface your home network is connected to. The bridge(4) is a layer 2 connection. Then simply assign an address from that same 192.168.1.0/24 network to a jail (just make sure you're not causing IP conflicts) and you should be good to go.
yzbythesea said:
but this is not recommended.
Click to expand...
Says who? Why wouldn't it be recommended? It's a perfectly valid set up. The only potential problem with this set up is when the uplink interface (which is connected to your home network) is a wireless interface. Then bridging is going to fail (not the bridge itself but the wireless interface effectively has to 'spoof' MAC addresses and most, if not all, wireless interfaces are going to refuse that).
 
SirDice said:
Bind the bridge to the same interface your home network is connected to
Click to expand...
Does this look good?

em0 is the interface my home network connects to
epair100 is the virtual interface that my Jail is using

Code: 
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=0
    ether 58:9c:fc:00:07:2e
    inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair100a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 1 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair100a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:myservice
    options=8<VLAN_MTU>
    ether 02:3e:f9:f0:74:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
That looks good. Assign a 192.168.1.0/24 address to the jail, set its default gateway to 192.168.1.1 just like everything else on your network. Also, with static addresses, don't forget about /etc/resolv.conf, this should probably also be set to 192.168.1.1 (I'm assuming your home internet router also acts as a DNS server, that's usually the case).
 
yzbythesea said:
Ah I meant to have them in the same home network. I will modify my FreeBSD server to have IP of 192.168.1.2 instead. So my home router is 192.168.1.1 and my server is 192.168.1.2.
Click to expand...
Hm, in that case, try this..
Code: 
cloned_interfaces="bridge0 epair0"
ifconfig_em0="up"
ifconfig_epair0a="up"
ifconfig_bridge0="inet 192.168.1.2/24 addm em0 addm epair0a up"
defaultrouter="192.168.1.1"
gateway_enable="YES"
And your jails will have IP addresses in the 192.168.1.0/24 subnet.
 
No need for the epair0 interface and addition to the bridge in rc.conf. It gets created and added by the exec.prestart in the jail's config, it's dynamically added and removed when the jail starts and stops.

Traditional jails (without VNET) are 'easier' to set up, but then you won't be able to do anything with the TCP/IP stack from within the jail. For a 'simple' jail connected to the same (home) network it usually suffices. In that case, set the IP address on the em0 interface and dispense the bridge configuration, don't need it. Bind the jail directly to em0.

Code: 
ifconfig_em0="inet 192.168.1.2 netmask 255.255.255.0"
In your jail configuration:
Code: 
myjail {
  {...}
  interface = em0;
  ipv4.addr = 192.168.1.3/24
  {...}
}
 
mumu said:
Hm, in that case, try this..
Code: 
cloned_interfaces="bridge0 epair0"
ifconfig_em0="up"
ifconfig_epair0a="up"
ifconfig_bridge0="inet 192.168.1.2/24 addm em0 addm epair0a up"
defaultrouter="192.168.1.1"
gateway_enable="YES"
And your jails will have IP addresses in the 192.168.1.0/24 subnet.
Click to expand...
Ah yes, this is the approach I am taking now. I does the dynamic adding/removing of epair via my Jail config. But we are on the same page.

SirDice said:
Traditional jails (without VNET) are 'easier' to set up, but then you won't be able to do anything with the TCP/IP stack from within the jail. For a 'simple' jail connected to the same (home) network it usually suffices. In that case, set the IP address on the em0 interface and dispense the bridge configuration, don't need it. Bind the jail directly to em0.
Click to expand...
This happens to solve my another use case where I do not need an isolated network stack but would like to have Jail to use unique IP so I can map a new DNS to it.

BTW I found out that I can also add my Bhyve tap0 interface to the same bridge bridge0 that has my Jail's epairs. I am using the pf to do a port forwarding (NAT) from my VM to the host IP for few service.

Thank you both!
 
You must log in or register to reply here.
Back
Top