[Solved] Bastille VNET jails can't be reached from host

J

jkp

I have a new 15.0 installation, where I have added Bastille and created two VNET jails:
# bastille create -VM testjail 15.0-RELEASE 192.168.3.200/24 re1
# bastille create -VM testjail1 15.0-RELEASE 192.168.3.201/24 re1
Click to expand...
Both have sshd enabled and a user added
# bastille sysrc testjail sshd_enable=YES
# bastille service testjail sshd start
# bastille cmd testjail adduser
# bastille sysrc testjail1 sshd_enable=YES
# bastille service testjail1 sshd start
# bastille cmd testjail1 adduser
Click to expand...
I can log into one of the jails using bastille console and ssh into the other. Works both ways. I can ssh into the jails from my workstation. What I can't do is ssh from the host and into the jails. The jails are on the same subnet as the host re1 interface (192.168.3.100).

ifconfig:
re1bridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:92:aa
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: e0a_testjail1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 11 priority 128 path cost 2000 vlan protocol 802.1q
member: e0a_testjail flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 7 priority 128 path cost 2000 vlan protocol 802.1q
member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 2 priority 128 path cost 20000 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_testjail: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vnet0 host interface for Bastille jail testjail
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:83:8a:da
hwaddr 58:9c:fc:10:6a:ca
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
e0a_testjail1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vnet0 host interface for Bastille jail testjail1
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:b5:9d:3a
hwaddr 58:9c:fc:10:36:f0
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Click to expand...

During installation of bastille, I made the following config changes:
rc.conf:
ifconfig_re1="inet 192.168.3.100/24"
defaultrouter="192.168.3.1"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"
Click to expand...
pf.conf:
## generated by bastille setup
int_if="re1"
localnet=$int_if:network

set block-policy return
scrub in all fragment reassemble
set skip on lo

table <jails> persist
nat on $int_if from <jails> to any -> ($int_if:0)
rdr-anchor "rdr/*"

pass in quick inet proto icmp icmp-type { echoreq, unreach }

pass out quick all keep state

pass in quick on $int_if proto tcp from any to any port ssh flags S/SA keep state

block in log (all) all
Click to expand...
devfs.rules:
[bastille_vnet=13]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path 'bpf*' unhide
Click to expand...
These are added to sysctl.conf:
# Bastille:
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_member=0 # Packet filter on the member interface
Click to expand...

What am I missing here? I have read the guide on bastillebsd.org and I can't see what's going wrong
 
The jails are on the same subnet as the host re1 interface (192.168.3.100).
Click to expand...

Perhaps try assigning this address to your re1bridge rather than re1.

ISTR reading that assigning addresses to bridge members has never worked reliably, though clearly it _has_ worked for most people most of the time. But it's certainly deprecated on 15.
 
Not sure I understand your entire configuration, but I have had success creating the bridge and assigning the host IP to the bridge itself (versus the physical interface). Then, when creating the jail with bastille, using the -B option with the bridge name.

rc.conf:
Code: 
cloned_interface="bridge0"
ifconfig_bridge0="inet 192.168.3.100 netmask 255.255.255.0 addm re1 up"
defaultrouter="192.168.3.1"

bastille command:

bastille create -V -B testjail 15.0-RELEASE 192.168.3.200 bridge0
 
I'm using Bastille version 1.4.1.260315 and it writes a bad /usr/local/bastille/jails/<your-jail-name>/jail.conf configuration, in case of VNET jails.

I had to patch it manually. IIRC, there is a fix in recent version of Bastille.

This is an example of working configuration, for a mine jail called "streaming":

Code: 
streaming {
  enforce_statfs = 2;
  devfs_ruleset = 4;
  exec.clean;
  exec.consolelog = /var/log/bastille/streaming_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = streaming;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/streaming/fstab;
  path = /usr/local/bastille/jails/streaming/root;
  securelevel = 2;
  osrelease = 14.3-RELEASE;


   vnet;
   vnet.interface = e0b_streaming;
   exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_streaming && ifconfig \${epair0%a}b up name e0b_streaming";
   exec.prestart += "ifconfig bridge0 addm e0a_streaming";
   exec.prestart += "ifconfig e0a_streaming description \"vnet0 host interface for Bastille jail streaming\"";
   exec.poststop += "ifconfig e0a_streaming destroy";
   allow.raw_sockets;
}

(edited) This is mine /etc/rc.conf

Code: 
hostname="server1.local.lan"
keymap="it.kbd"

moused_nondefault_enable="NO"


sshd_enable="YES"
linux_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

# The interface where the server will receive the DHCP IP
# from modem/gateway, i.e. the external interface
ifconfig_bge3="DHCP"

# Create a bridge with the interfaces to use for DHCP connections.
# bridge0 is used both for DHCP connections and for connecting VNET jails.
gateway_enable="YES"
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0="addm bge0 addm bge1 addm bge2 addm tap0 addm tap1 addm tap3 addm tap2 up tap3 up"
ifconfig_bridge0_alias0="inet 192.168.2.1/24"
ifconfig_bge0="up"
ifconfig_bge1="up"
ifconfig_bge2="up"

dhcpd_ifaces="bridge0"

rcshutdown_timeout="900"
microcode_update_enable="YES"
sshd_dsa_enable="no"
sshd_ecdsa_enable="no"
sshd_ed25519_enable="YES"
sshd_rsa_enable="YES"
bastille_enable="YES"

unbound_enable="YES"

ntpd_enable="YES"
ntpd_sync_on_start="YES"

salt_master_enable="YES"
node_exporter_enable="yes"

ng_netflow_enable="YES"

ntpdate_enable="YES"
dhcpd_enable="YES"
 
putney said:
Perhaps try assigning this address to your re1bridge rather than re1.

ISTR reading that assigning addresses to bridge members has never worked reliably, though clearly it _has_ worked for most people most of the time. But it's certainly deprecated on 15.
Click to expand...
I can try that. I haven't heard that assigning addresses to bridge members was a no-go.
 
OpenFreeNet said:
I'm using Bastille version 1.4.1.260315 and it writes a bad /usr/local/bastille/jails/<your-jail-name>/jail.conf configuration, in case of VNET jails.

I had to patch it manually. IIRC, there is a fix in recent version of Bastille.

This is an example of working configuration, for a mine jail called "streaming":

Code: 
streaming {
  enforce_statfs = 2;
  devfs_ruleset = 4;
  exec.clean;
  exec.consolelog = /var/log/bastille/streaming_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = streaming;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/streaming/fstab;
  path = /usr/local/bastille/jails/streaming/root;
  securelevel = 2;
  osrelease = 14.3-RELEASE;


   vnet;
   vnet.interface = e0b_streaming;
   exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_streaming && ifconfig \${epair0%a}b up name e0b_streaming";
   exec.prestart += "ifconfig bridge0 addm e0a_streaming";
   exec.prestart += "ifconfig e0a_streaming description \"vnet0 host interface for Bastille jail streaming\"";
   exec.poststop += "ifconfig e0a_streaming destroy";
   allow.raw_sockets;
}

(edited) This is mine /etc/rc.conf

Code: 
hostname="server1.local.lan"
keymap="it.kbd"

moused_nondefault_enable="NO"


sshd_enable="YES"
linux_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

# The interface where the server will receive the DHCP IP
# from modem/gateway, i.e. the external interface
ifconfig_bge3="DHCP"

# Create a bridge with the interfaces to use for DHCP connections.
# bridge0 is used both for DHCP connections and for connecting VNET jails.
gateway_enable="YES"
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0="addm bge0 addm bge1 addm bge2 addm tap0 addm tap1 addm tap3 addm tap2 up tap3 up"
ifconfig_bridge0_alias0="inet 192.168.2.1/24"
ifconfig_bge0="up"
ifconfig_bge1="up"
ifconfig_bge2="up"

dhcpd_ifaces="bridge0"

rcshutdown_timeout="900"
microcode_update_enable="YES"
sshd_dsa_enable="no"
sshd_ecdsa_enable="no"
sshd_ed25519_enable="YES"
sshd_rsa_enable="YES"
bastille_enable="YES"

unbound_enable="YES"

ntpd_enable="YES"
ntpd_sync_on_start="YES"

salt_master_enable="YES"
node_exporter_enable="yes"

ng_netflow_enable="YES"

ntpdate_enable="YES"
dhcpd_enable="YES"
Click to expand...
I think I have the latest version of Bastille, but I'll check later today (server is @ home) and try to assign the IP to the bridge instead of the interface.
 
OpenFreeNet said:
I'm using Bastille version 1.4.1.260315 and it writes a bad /usr/local/bastille/jails/<your-jail-name>/jail.conf configuration, in case of VNET jails.

I had to patch it manually. IIRC, there is a fix in recent version of Bastille.

This is an example of working configuration, for a mine jail called "streaming":

Code: 
streaming {
  enforce_statfs = 2;
  devfs_ruleset = 4;
  exec.clean;
  exec.consolelog = /var/log/bastille/streaming_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = streaming;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/streaming/fstab;
  path = /usr/local/bastille/jails/streaming/root;
  securelevel = 2;
  osrelease = 14.3-RELEASE;


   vnet;
   vnet.interface = e0b_streaming;
   exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_streaming && ifconfig \${epair0%a}b up name e0b_streaming";
   exec.prestart += "ifconfig bridge0 addm e0a_streaming";
   exec.prestart += "ifconfig e0a_streaming description \"vnet0 host interface for Bastille jail streaming\"";
   exec.poststop += "ifconfig e0a_streaming destroy";
   allow.raw_sockets;
}

(edited) This is mine /etc/rc.conf

Code: 
hostname="server1.local.lan"
keymap="it.kbd"

moused_nondefault_enable="NO"


sshd_enable="YES"
linux_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

# The interface where the server will receive the DHCP IP
# from modem/gateway, i.e. the external interface
ifconfig_bge3="DHCP"

# Create a bridge with the interfaces to use for DHCP connections.
# bridge0 is used both for DHCP connections and for connecting VNET jails.
gateway_enable="YES"
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0="addm bge0 addm bge1 addm bge2 addm tap0 addm tap1 addm tap3 addm tap2 up tap3 up"
ifconfig_bridge0_alias0="inet 192.168.2.1/24"
ifconfig_bge0="up"
ifconfig_bge1="up"
ifconfig_bge2="up"

dhcpd_ifaces="bridge0"

rcshutdown_timeout="900"
microcode_update_enable="YES"
sshd_dsa_enable="no"
sshd_ecdsa_enable="no"
sshd_ed25519_enable="YES"
sshd_rsa_enable="YES"
bastille_enable="YES"

unbound_enable="YES"

ntpd_enable="YES"
ntpd_sync_on_start="YES"

salt_master_enable="YES"
node_exporter_enable="yes"

ng_netflow_enable="YES"

ntpdate_enable="YES"
dhcpd_enable="YES"
Click to expand...
This is because you are using a bridge here, and your IP is assigned there. All the FreeBSD network stuff will tell you that this is the correct way. You shouldn’t assign an IP to the member. It should go on the bridge.
 
I have a similar problem, not with Bastille, with iocage. I have been reconfiguring my jail hosts today to assign the IP address to the bridge instead of the interface and I have noticed a pattern that may/may not be relevant to you.

On machines that have been installed fresh with 15.0-RELEASE, assigning the IP address to the bridge works OK. My jail hosts are connected to a untagged VLAN ports on their switches. I am now using if_vlan_load="YES" in /boot/loader.conf which seems to make bridge IP work well for me with tagged VLANs present on the ports.

On jail hosts that were built years ago on earlier FreeBSD versions that have since upgraded to 15.0, I have not been successful at moving to them to bridge IP. They were built following iocage documentation with IP on the network interface. With these old machines reconfigured for bridge IP assignment, the jails cannot resolve DNS queries.

With PF disabled for testing, from the old jail hosts with new bridge IP configuration, I can:
1. Ping the default gateway for the untagged VLAN, 10.26.4.254
2. Ping Cloudflare DNS 1.1.1.1
3. Ping the IP addresses of all of the started vnet jails on the same VLAN as the jail host 10.26.4.x
4. Resolve pkg.freebsd.org using dig, 151.101.1.241, 151.101.65.241, 151.101.129.241, 151.101.193.241
5. Resolve the IP addresses of all vnet jails on the jail host
All appears to be OK from the jail host.

However. from each vnet jail on that host, I can:
6. Ping the default gateway for the untagged VLAN, 10.26.4.254
7. Ping Cloudflare DNS 1.1.1.1
8. Ping the IP addresses of all of the started vnet jails on the same VLAN as the jail host 10.26.4.x
9. All DNS queries fail!

I cannot resolve DNS from any jail that is on a 15.0 jail host that was previously upgraded from 14.3 after switching to bridge IP configuration. Today I created new thick jails running 13.5, 14.3, 14.4 and 15.0 on the same old jail hosts and each test jail has the same DNS resolution problem.

I created a new thick 15.0 vnet test jail on a fully working native 15.0 jail host running bridge IP assignment. This jail works fine on that host. After using zfs send/recv to transfer it to an old upgraded 14.3 to 15.0 jail host, the jail can no longer resolve. I have reset the vnet MAC address for the jail before starting it at it's new location. It can ping IPv4 addresses anywhere as its default gateway is operational. There is no firewall running on the jail host or externally that blocks port 53.

I shutdown the 15.0 test jail to rename it. I deleted all of its existing snapshots then created a new recursive snapshot before sending it back to the original jail host it was created on. When that had completed, I reset the jail's vnet0 MAC address and started the jail. It runs perfectly with a new name back on the orinating jail host. It can resolve DNS and install packages successfully.

Strangely, if I uncomment the former working config on the old jail hosts in /etc/rc.conf and comment out the new bridge IP config, the jails still do not work properly after a reboot. I cannot easily roll back ZFS as some jails have had data changes since noticing this problem. As they will work on the freshly installed 15.0 jail hosts, I am transferring them to unaffected hosts with the new bridge IP config. I will wipe the disks on the old hosts for fresh 15.0 installs, then move the jails back.

I have just had a thought. I have not checked if there is a difference between sysctl.conf on unaffected and affected jail hosts. I will try this later when the jail transfers finish.
 
Setting the ip on the bridge worked. I switched to pre-creating the bridge in host rc.conf and setting the ip on the bridge instead of the interface. Creating the jail with -B and using the pre-created bridge works. I can ping and ssh to the jail now.

Thanks to all that replied
 
You must log in or register to reply here.
Back
Top