Solved jail networking

T

tOsYZYny

Can I assign an IP to an epair interface for a jail that would be on the same subnet as my LAN?

I have a router jail with 2 interfaces (wired and wan) and a 'workstation' jail which has an epair to the host system and an epair to the router jail. I'd like to assign the workstation an IP on the wired subnet so that I could talk to other clients on that network and vice versa.

I am thinking that perhaps it should not be a vnet jail:
docs.freebsd.org

Chapter 17. Jails and Containers

Jails improve on the concept of the traditional chroot environment in several ways
docs.freebsd.org docs.freebsd.org

Then, I should assign the IP address and interface that I want and it *should* magically work. The interface in my situation would be wired. Would that pose any problems that the wired interface is assigned to a jail?
 
tOsYZYny said:
Can I assign an IP to an epair interface for a jail that would be on the same subnet as my LAN?
Click to expand...
If you bridge(4) it to your LAN interface, yes. Things might get a little dicey if your LAN interface is wireless though, but as long as it's wired it should work fine.

tOsYZYny said:
I am thinking that perhaps it should not be a vnet jail:
Click to expand...
A non-VNET jail, in other words, a 'classic' jail can be bound directly to the LAN interface. But you obviously lose the ability to control the network interface from within the jail. Simpler solution. VNET jail could work just as well, if you bridge(4) the epair interface with the LAN interface.
 
Hmm, I think my setup is probably what is causing it to not work.

My base system has assigned all physical networking interfaces to the router jail: wired and wan. The base system has a patch cable to the router so it can get to the Internet (for updates).

My workstation (which I want to be on the LAN for sharing media) also had a patch cable until recently. Instead this is what I changed, but resulted in no networking which I think is because the wired interface is owned by the jail?

1. create bridge, bridge0
2. add wired to bridge0
3. create epair1
4. assign epair1b to workstation with an IP on the same subnet as the wired interface
5. assign epair1b to the bridge

I have setup allow.raw_sockets, but I cannot ping the wired interface. I was doing a tcpdump on wired and pflog1 (the interface I'm logging blocked traffic) and there was nothing. I suspect that because the bridge0 is owned by the base system and not the router jail, that perhaps that is messing it up.

This was the document I was following:
docs.freebsd.org

Chapter 17. Jails and Containers

Jails improve on the concept of the traditional chroot environment in several ways
docs.freebsd.org docs.freebsd.org

snippet of jail.conf
Code: 
   allow.raw_sockets;
    vnet;
    vnet.interface = epair1b;

    exec.prestart    += "ifconfig epair1 create";
    exec.prestart    += "ifconfig epair1a up";
    exec.prestart    += "ifconfig bridge create";
    exec.prestart    += "ifconfig bridge0 addm wired up";
    exec.prestart    += "ifconfig bridge0 addm epair1a up";

    exec.start     += "ifconfig epair1b $workstation_ip/24 up";
    exec.start     += "route add default $wired_ip";

    exec.poststop  += "ifconfig epair1a destroy";
    exec.poststop  += "ifconfig bridge0 deletem epair1a";
    exec.poststop  += "ifconfig bridge0 destroy";
 
I also tried assigning the bridge (an IP on the same subnet as the wired interface) and the "a" side of the epair interface to the router jail, but to no avail. When trying to ping the gateway (wired interface IP) from the workstation, I get host is down.

As I think about it now some more, I am thinking that whenever I assign it to the jail that I probably need to redo the networking as the IP would be lost, round 300.
 
I think I got it, I had to do all of that setup in the router jail.

router.conf
Code: 
exec.created   += "ifconfig bridge0 vnet router";
exec.created   += "ifconfig epair1a vnet router";
exec.created   += "jexec router ifconfig epair1a up"
exec.created    += "jexec ifconfig bridge0 $bridge_ip/24 up";
exec.created    += "jexec ifconfig bridge0 addm wired up";
exec.created    += "jexec ifconfig bridge0 addm epair1a up";

workstation.conf
Code: 
vnet;
vnet.interface = epair1b;

exec.prestart    += "ifconfig epair1 create";
exec.prestart    += "ifconfig epair1a up";
exec.prestart    += "ifconfig bridge create";

exec.start     += "ifconfig epair1b $workstation_ip/24 up";
exec.start     += "route add default $gateway";

exec.poststop  += "ifconfig epair1a destroy";
exec.poststop  += "ifconfig bridge0 deletem epair1a";
exec.poststop  += "ifconfig bridge0 destroy";
 
I've had the same experience testing the same configuration from the manual for the last few day. Even tried multiple other configurations from here and there with no success at all.

Another issue I've had, which has surprised me is the "Permission denied" when issuing ping. I've declared the allow.raw_sockets directive in the jail configuration, since issuing it in sysctl.conf has been deprecated.

System at hand is a Lenovo T430 laptop with wired and wireless connectivity.

/etc/rc.conf

Code: 
hostname="BSDTest"
keymap="es.acc.kbd"
ifconfig_em0="DHCP"
sshd_enable="YES"
moused_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"
dbus_enable="YES"
slim_enable="YES"
kld_list="i915kms"
defaultrouter="192.168.1.1"
wpa_supplicant_program="/usr/local/sbin/wpa_supplicant"
wlans_iwn0="wlan0"
create_args_wlan0="country ES regdomain etsi"
ifconfig_wlan0="WPA DHCP"
firewall_enable="YES"
firewall_quiet="YES"
firewall_type="workstation"
firewall_logdeny="YES"
firewall_allowservices="any"
firewall_myservices="22/tcp 80/tcp 443/tcp"
sendmail_submit_enable="NO"
sendmail_msp_queue_enable="NO"
sendmail_enable="NO"
sendmail_outbound_enable="NO"
bastille_enable="YES"
bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot"
linux_enable="YES"
webcamd_enable="YES"
webcamd_0_flags="-d ugen1.6 -N Chicony-Electronics-Co---Ltd--Integrated-Camera -S unknown -M 0"
jail_enable="YES"
jail_parallel_start="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"

Notes:

1.- Yes ifconfig_em0="DHCP" is not present. The system gets it assigned automatically from this network.
2.- No, no bastille jail nor iocage ones are active or running in this system.

ifconfig on the host

Code: 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 28:d2:44:43:79:e0
        inet 192.168.1.197 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a4:4e:31:dc:39:24
        inet 192.168.1.190 netmask 0xffffff00 broadcast 192.168.1.255
        groups: wlan
        ssid MIWIFI_dbdG channel 36 (5180 MHz 11a ht/40+) bssid 44:d4:53:4e:57:55
        regdomain ETSI country ES authmode WPA2/802.11i privacy ON
        deftxkey UNDEF AES-CCM 2:128-bit txpower 17 bmiss 10 mcastrate 6
        mgmtrate 6 scanvalid 60 ampdulimit 64k ampdudensity 4 -amsdutx amsdurx
        shortgi -stbc -ldpc -uapsd wme roaming MANUAL
        parent interface: iwn0
        media: IEEE 802.11 Wireless Ethernet MCS mode 11na
        status: associated
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:cd:76
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 02:15:e0:ec:01:00
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

The jail configuration is set in /etc/jail.conf and reads as follows:

Code: 
jailone {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets = 1;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # PATH/HOSTNAME
  path = "/usr/local/jails/containers/${name}";
  host.hostname = "${name}";

  # VNET/VIMAGE
  vnet;
  vnet.interface = "${epair}b";

  # NETWORKS/INTERFACES
  $id = "154";
  $ip = "10.0.1.${id}/24";
  $gateway = "10.0.1.1";
  $bridge = "bridge0";
  $epair = "epair${id}";

  # ADD TO bridge INTERFACE
  exec.prestart += "ifconfig ${epair} create up";
  exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
  exec.start    += "ifconfig ${epair}b ${ip} up";
  exec.start    += "route add default ${gateway}";
  exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "ifconfig ${epair}a destroy";
}

Other config files, just in case something is very much needed.

/etc/sysctl.conf

Code: 
# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12
vfs.usermount=1
net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0

/etc/devfs.rules

Code: 
[devfsrules_jails=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide

Things I've tried:

0.- Default settings as in the manual.
1.- With and without net.inet.ip.forwarding in /etc/sysctl.conf.
2.- With and without setting net.link.bridge directives in /etc/devfs.rules.
3.- Assigning and IP address to the bridge and/or the host epair and leaving them empty relying on the /etc/jail.conf configuration.
4.- Configuring the networking jail settings on the same LAN as the host as well as a different range and everything in between.
5.- Set up the bridge with an IP and none. Same as in point 4.
6.- Tested with iocage VNET config settings from the official documentation.

I've also tested with a few settings from this document: https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf , with no success.

To summarize, I've tried anything that has come up in my head with a negative result every time. I've also tested this on a VM on VirtualBox with promiscuous mode on and off.

My impression is something is off on the epair side of things. I'm able to ping the bridge when an IP is assigned to it, as well as to the host's interface, but the jail doesn't get newtorking at all. Can't ping the gateway whatever config I use, nor have dns resolution or anything of the sort.

Bug or my bad?
 
With my previous post, I was able to get my jail on the same subnet as my wired interface and everything works as expected.

1. ping should work provided you have correct permissions which allow raw sockets grants. Do you happen to have firewall rules on the jail or host that might be preventing ping from working?
2. I think you might be missing the other interface in the bridge, you have the epaira interface, but I think you should also have either em0 or wlan0 in the bridge (I would suggest em0).

Once you add em0 to the bridge, it should be as if the epairb is on the same network as em0 and you should be able to ping it.
 
You must log in or register to reply here.
Back
Top