Solved IOCAGE on a VPS

Setting up an IOCAGE jail on my home system involves adding these lines to /etc/rc/conf

Code:
ifconfig_em0="DHCP"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"

What do I do on my VPS?

Code:
ifconfig_vtnet0="DHCP"
=========>cloned_interfaces="bridge0" ???????????/
ifconfig_vtnet0_ipv6="inet6 accept_rtadv"

I guess I need an 'ifconfig_bridge0'

Any suggestions?

I'm really out of my depth here.
 
Looks like I have screwed up the interface and can't even ping 8.8.8.8 from within my host.

After removing the lines I added and rebooted I can now ping, but am no nearer to getting this working.

This is what I have on my home server:
Code:
ifconfig_em0="DHCP"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"

The IOCAGE docs do not mentioned cloned interaces, but this entry was suggested here and it has been working fine.

Do I need to add the same lines with appropriate changes to my VPS?

On my VPS I have vtnet0 instead of em0.
There is also ifconfig_vtnet0_ipv6="inet6 accept_rtadv"
 
You cannot bridge your jails with your external interface on your VPS. You do need a bridge (to attach your jails to) but it'll be disconnected from your external interface, so this "network" will be entirely local. Then you set up NAT and such so your jails can connect to the outside world.
 
You cannot bridge your jails with your external interface on your VPS. You do need a bridge (to attach your jails to) but it'll be disconnected from your external interface, so this "network" will be entirely local. Then you set up NAT and such so your jails can connect to the outside world.
Looks like I am out of my depth trying to achieve what I want.

I'll quote what GROK advises and then maybe someone can correct any mistakes in the suggested solution..
Code:
ifconfig bridge create
ifconfig bridge0 name br0
ifconfig br0 inet YOUR_PUBLIC_IP netmask 255.255.255.0 up addm vtnet0
ifconfig br0 up

sysrc cloned_interfaces="br0"
sysrc ifconfig_br0="inet YOUR_PUBLIC_IP netmask 255.255.255.0 up addm vtnet0"
sysrc gateway_enable="YES"

ee /etc/pf.conf

ext_if = "vtnet0"
int_net = "10.0.0.0/24"

set skip on lo0

# NAT for jails
nat on $ext_if from $int_net to any -> ($ext_if)

# Allow jails to reach the internet
pass out quick on $ext_if inet from $int_net to any keep state

# Allow SSH + HTTP/HTTPS to the host (add more ports as needed)
pass in on $ext_if inet proto tcp from any to ($ext_if) port { 22, 80, 443 } keep state

# Default policy
block in log all

# Reload network
service netif restart
service routing restart

# Enable and start PF
sysrc pf_enable="YES"
service pf start
pfctl -f /etc/pf.conf

iocage create -n testjail -r 14.2-RELEASE \
  vnet=on \
  ip4_addr="br0|10.0.0.10/24" \
  defaultrouter="10.0.0.1" \
  boot=on
 
  iocage start testjail
iocage console testjail

Is this any good to get me started?

It looks as though GROK has not yet learned about how to create HEREDOCs :)
 
Is this any good to get me started?
Nope. Grok seems to add the external interface to the bridge, that's a mistake. Well, in this case it is, it is appropriate in other situations.

Doing this from the top of my head, might not be entirely correct:
Code:
gateway_enable="YES" # You need routing enabled
ifconfig_vtnet0="DHCP"  # I'm assuming that's your VPS 'external' interface.
cloned_interfaces="bridge0" # You do want to create a bridge, so you can have an 'internal' network to connect your jails to
ifconfig_bridge0="inet 10.0.0.1 netmask 255.255.255.0" # giving the bridge 10.0.0.1/24

pf.conf:
Code:
ext_if="vtnet0"
int_if="bridge0"

nat on $ext_if from $int_if:network to any -> ($ext_if)

Then connect your jails to bridge0, with or without VNET, doesn't matter, both should work. Pick an address in the 10.0.0.0/24 range for the jail, so it's in the same subnet as the bridge interface, for VNET jails use 10.0.0.1 (the bridge's address) as the gateway address.
 
Nope. Grok seems to add the external interface to the bridge, that's a mistake. Well, in this case it is, it is appropriate in other situations.

Doing this from the top of my head, might not be entirely correct:
Code:
gateway_enable="YES" # You need routing enabled
ifconfig_vtnet0="DHCP"  # I'm assuming that's your VPS 'external' interface.
cloned_interfaces="bridge0" # You do want to create a bridge, so you can have an 'internal' network to connect your jails to
ifconfig_bridge0="inet 10.0.0.1 netmask 255.255.255.0" # giving the bridge 10.0.0.1/24

pf.conf:
Code:
ext_if="vtnet0"
int_if="bridge0"

nat on $ext_if from $int_if:network to any -> ($ext_if)

Then connect your jails to bridge0, with or without VNET, doesn't matter, both should work. Pick an address in the 10.0.0.0/24 range for the jail, so it's in the same subnet as the bridge interface, for VNET jails use 10.0.0.1 (the bridge's address) as the gateway address.
I need to do this in simple stages because somehow I ended screwing up the network and couldn't even ping the host.

I noticed that when I added gateway_enable="YES" and then ran service netif restart a msg came up saying IP address (***.***.***.***) was deleted and could no longer ping my VPS.
 
Not knowing anything about VNET, I thought I would look up some info via GROK:-

FreeBSD VNET refers to the Virtual Network Stack (also called VIMAGE).
It gives each jail its own complete, isolated network stack — its own interfaces, routing table, ARP cache, firewall (PF/IPFW), and sockets. The jail behaves almost like a separate machine.

VNET vs normal (shared) jail networking​

FeatureShared network stackVNET
Network isolationLowHigh
Own interfacesNo (uses host IPs)Yes
Own routing tableNoYes
Own firewallNo (shares host PF)Yes
ComplexitySimpleHigher
Performance overheadNoneSmall
Best forSimple servicesFull isolation, VPN, routers, etc.


Basic requirements​

VNET is enabled by default in modern FreeBSD (13+ / 14 / 15):
Bash

Code:
[COLOR=rgb(126, 178, 51)]sysctl[/COLOR][COLOR=rgb(164, 65, 133)] kern.features.vimage[/COLOR][/COLOR]
[COLOR=rgb(0, 35, 57)][I][COLOR=rgb(53, 123, 66)]# should return 1[/COLOR][/I]


Load the needed modules:

Bash

[COLOR=rgb(126, 178, 51)]sysrc[/COLOR][COLOR=rgb(164, 65, 133)] kld_list+="if_bridge if_epair"[/COLOR]
How VNET jails are usually connected

Create a bridge on the host (connected to the physical/VPS interface, often vtnet0).
Create an epair (virtual Ethernet cable) for each jail.
One end of the epair stays on the host (added to the bridge).
The other end is moved into the jail (vnet.interface).

Example simplified jail.conf snippet:
confweb {
path = "/jails/web";
host.hostname = "web.example.com";
vnet;
vnet.interface = "epair0b";

exec.prestart = "ifconfig epair0 create";
exec.prestart += "ifconfig bridge0 addm epair0a";
exec.prestart += "ifconfig epair0a up";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair0a destroy";

mount.devfs;
allow.raw_sockets;
}
Code:
[COLOR=rgb(126, 178, 51)]kldload[/COLOR][COLOR=rgb(164, 65, 133)] if_bridge if_epair[/COLOR]



How VNET jails are usually connected​


  1. Create a bridge on the host (connected to the physical/VPS interface, often vtnet0).
  2. Create an epair (virtual Ethernet cable) for each jail.
  3. One end of the epair stays on the host (added to the bridge).
  4. The other end is moved into the jail (vnet.interface).
Example simplified jail.conf snippet:


conf

web {
path = "/jails/web";
host.hostname = "web.example.com";
vnet;
vnet.interface = "epair0b";

exec.prestart = "ifconfig epair0 create";
exec.prestart += "ifconfig bridge0 addm epair0a";
exec.prestart += "ifconfig epair0a up";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair0a destroy";

mount.devfs;
allow.raw_sockets;
}


Inside the jail you then configure the interface normally (ifconfig_epair0b=... in the jail’s /etc/rc.conf).


Note about​


On most FreeBSD VPS (KVM/QEMU), the physical network interface is called vtnet0 (VirtIO network driver).This is different from VNET (the jail feature).



Would you like a complete working example for:

  • Simple VNET jail with bridge + NAT, or
  • VNET jail bridged directly to the public interface (vtnet0)?


Anyone care to tell me which of those I want since this is all gobbledigook to me?
 
What you should do is ask Grok to explain things to you piece by piece. Start at the top and just say you don't understand or know how something works. Then work your way down.
I like Grok but tend to use Claude for computer stuff.
 
More GROK speak:-

Code:
FreeBSD VNET + NAT setup (private network for jails + NAT to the public interface).
This is the most common and practical configuration on a VPS (one public IP).
1. Host preparation
Bash# Load required modules
sysrc kld_list+="if_bridge if_epair"
kldload if_bridge if_epair

# Enable IP forwarding
sysrc gateway_enable="YES"
sysctl net.inet.ip.forwarding=1
2. Create the bridge and private network
Bash# Create bridge
ifconfig bridge0 create
sysrc cloned_interfaces+="bridge0"
sysrc ifconfig_bridge0="inet 10.0.0.1/24 up"

Any comments before I try this?
 
That looks good. Doesn't do much besides creating the bridge interface, but it's a good start.
At least it comes up after a reboot.

Originally when starting my TEST IOCAGE jail it failed immediately, this time it seems to start but fails with
Code:
# iocage start TEST
No default gateway found for ipv6.
* Starting TEST
  + Started OK
  + Using devfs_ruleset: 1000 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
  + Acquiring DHCP address: FAILED, address received: ERROR, check jail logs

Stopped TEST due to DHCP failure

The jail was created much earlier using:-

iocage create -r latest -n TEST vnet=on dhcp=on
 
The issue with VNET jails on a VPS is that you won’t really have any IPs to give them. A VNET jail is like a second instance with its own IP etc…

You want a NAT jail. I’ve used Iocage in the past, but have since switched to Bastille, which made things a lot easier for me. Especially with Iocage not really being updated with new features. It’s being maintained somewhat, but only bug fixes and important stuff as I can tell.

Nevertheless, what you want is a NAT jail, not a VNET unless your VPS hosted is giving you a bunch of public IPs.
 
The issue with VNET jails on a VPS is that you won’t really have any IPs to give them. A VNET jail is like a second instance with its own IP etc…

You want a NAT jail. I’ve used Iocage in the past, but have since switched to Bastille, which made things a lot easier for me. Especially with Iocage not really being updated with new features. It’s being maintained somewhat, but only bug fixes and important stuff as I can tell.

Nevertheless, what you want is a NAT jail, not a VNET unless your VPS hosted is giving you a bunch of public IPs.
I use IOCAGE at home on an internal network and IP addresses are assigned to 192.168.0.*.

Maybe it doesn't need updates because it isn't broken.

There is a github page with some recent updates:-


All this stuff is way out of my depth and I'm only going by what I can glean from various searches.
 
Code:
Stopped jail due to DHCP failure"Stopped jail due to DHCP failure" is a common FreeBSD jail error.
It means the jail was configured to get its IP via DHCP, but dhclient failed to obtain a lease, so the jail was stopped.
Why it happens with VNET
In a VNET jail the network interface (usually epairXb) is created only when the jail starts.
If the jail’s /etc/rc.conf contains something like:
confifconfig_epair0b="DHCP"
then dhclient runs, finds no DHCP server (or the interface isn’t ready yet), and the jail startup aborts.
This is especially common when:

You bridged the jail directly to the public interface (vtnet0)
Your VPS provider does not give DHCP leases to extra MAC addresses
You are using a private bridge + NAT (there is normally no DHCP server on 10.0.0.0/24)

Code:
Solution: Use a static IP instead of DHCP
Inside the jail (or in the jail template):
Bash# Remove DHCP
sysrc -x ifconfig_epair0b

# Set a static IP
sysrc ifconfig_epair0b="inet 10.0.0.10/24"          # private NAT version
# or
sysrc ifconfig_epair0b="inet 203.0.113.20/24"       # public IP version

sysrc defaultrouter="10.0.0.1"                      # or your real gateway
Also make sure DNS is set:
Bashecho "nameserver 1.1.1.1" > /etc/resolv.conf
Then restart the jail:
Bashservice jail restart web
Quick check
 
When adding the following to /etc/rc.conf
Code:
cloned_interfaces="bridge0"
ifconfig_bridge0="addm vtnet0 up"
ifconfig_vtnet0="up"

and running service netif restart the system hangs.

Is this to be expected?

This is on my VPS host using a static IP address.
 
After making the changes above my ssh session becomes disconnected and subsequently I can no longer use ssh.

When logging in via VPS console, I see the changes were made but I can no longer ping 8.8.8.8.

Running route add default 168.235.83.1 (my defaultrouter) errors with fib 0: invalid argument.

Can anyone explain what is going wrong?
 
Please show us
  • /etc/rc.conf
  • ifconfig
  • netstat -rn
It's difficult to that since I don't see any way to copy from the VPS console. I can't ssh into the system.

I followed what GROK suggested:-

Code:
# Physical interface – no IP, just up
ifconfig_vtnet0="up"

# Bridge that holds the host IP and connects jails
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 203.0.113.10/24 addm vtnet0 up"

# Gateway stays the same
defaultrouter="203.0.113.1"

although I changed the values above to my own host.

ifconfig shows vtnet0 lo bridge0 but only lo has an IP address assigned.

netstat -rn has link#2 under Gateway for 7 entries. Nothing else.
 
Please show us
  • /etc/rc.conf
  • ifconfig
  • netstat -rn
This is after I have managed to fix things and log back in.

Code:
gateway_enable="YES"
ifconfig_vtnet0="168.235.83.112/24"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.0.1/24"



#ifconfig_vtnet0="up"
#cloned_interfaces="bridge0"

#ifconfig_vtnet0="168.235.83.112/24"
#ifconfig_bridge0="inet 168.235.83.112/24 addm vnet0 up"

defaultrouter="168.235.83.1"

#ifconfig_bridge0="addm vtnet0 up"          # ← change vtnet0 to your real interface

sshd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"
tuptime_enable="YES"
Code:
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=ec079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS>
        ether fa:16:3e:c5:ff:08
        inet 168.235.83.112 netmask 0xffffff00 broadcast 168.235.83.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=10<VLAN_HWTAGGING>
        ether 58:9c:fc:10:a3:66
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        bridge flags=0<>
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
Code:
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            168.235.83.1       UGS          vtnet0
10.0.0.0/24        link#3             U           bridge0
10.0.0.1           link#2             UHS             lo0
127.0.0.1          link#2             UH              lo0
168.235.83.0/24    link#1             U            vtnet0
168.235.83.112     link#2             UHS             lo0

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#2                        URS             lo0
::1                               link#2                        UHS             lo0
::ffff:0.0.0.0/96                 link#2                        URS             lo0
fe80::%lo0/10                     link#2                        URS             lo0
fe80::%lo0/64                     link#2                        U               lo0
fe80::1%lo0                       link#2                        UHS             lo0
ff02::/16                         link#2                        URS             lo0
 
Some progress seems to be being made but I can't start my jail.

# iocage start TEST
No default gateway found for ipv6.
* Starting TEST
+ Started OK
+ Using devfs_ruleset: 1000 (iocage generated default)
+ Configuring VNET OK
+ Using IP options: vnet
+ Starting services OK
+ Executing poststart OK
+ Acquiring DHCP address: FAILED, address received: ERROR, check jail logs

Stopped TEST due to DHCP failure


This is from /var/log/messages in my TEST jail.
Jul 27 07:46:52 kernel: epair0a: Ethernet address: 58:9c:fc:10:48:7c
Jul 27 07:46:52 kerrnel: epair0b: Ethernet address: 58:9c:fc:10:d5:2c
Jul 27 07:46:52 kernel: epair0a: link state changed to UP
Jul 27 07:46:52 kernel: epair0b: link state changed to UP
Jul 27 07:46:52 kernel: epair0a: changing name to 'vnet0.3'
Jul 27 07:46:52 kernel: bridge0: link state changed to UP
Jul 27 07:46:52 kernel: vnet0.3: promiscuous mode enabled
Jul 27 07:46:53 kernel: lo0: link state changed to UP
Jul 27 07:48:25 kernel: vnet0.3: link state changed to DOWN
Jul 27 07:48:25 kernel: epair0b: link state changed to DOWN
Jul 27 07:48:25 kernel: bridge0: link state changed to DOWN


Can someone explain where these epair0* entries come from and what is missing?
 
Back
Top