Bastille VNET jails can't be reached from host

[jkp]

I have a new 15.0 installation, where I have added Bastille and created two VNET jails:

# bastille create -VM testjail 15.0-RELEASE 192.168.3.200/24 re1
# bastille create -VM testjail1 15.0-RELEASE 192.168.3.201/24 re1

Both have sshd enabled and a user added

# bastille sysrc testjail sshd_enable=YES
# bastille service testjail sshd start
# bastille cmd testjail adduser
# bastille sysrc testjail1 sshd_enable=YES
# bastille service testjail1 sshd start
# bastille cmd testjail1 adduser

I can log into one of the jails using bastille console and ssh into the other. Works both ways. I can ssh into the jails from my workstation. What I can't do is ssh from the host and into the jails. The jails are on the same subnet as the host re1 interface (192.168.3.100).

ifconfig:

re1bridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:92:aa
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: e0a_testjail1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 11 priority 128 path cost 2000 vlan protocol 802.1q
member: e0a_testjail flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 7 priority 128 path cost 2000 vlan protocol 802.1q
member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 2 priority 128 path cost 20000 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_testjail: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vnet0 host interface for Bastille jail testjail
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:83:8a:da
hwaddr 58:9c:fc:10:6a:ca
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
e0a_testjail1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vnet0 host interface for Bastille jail testjail1
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:b5:9d:3a
hwaddr 58:9c:fc:10:36:f0
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

During installation of bastille, I made the following config changes:
rc.conf:

ifconfig_re1="inet 192.168.3.100/24"
defaultrouter="192.168.3.1"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"

pf.conf:

## generated by bastille setup
int_if="re1"
localnet=$int_if:network

set block-policy return
scrub in all fragment reassemble
set skip on lo

table <jails> persist
nat on $int_if from <jails> to any -> ($int_if:0)
rdr-anchor "rdr/*"

pass in quick inet proto icmp icmp-type { echoreq, unreach }

pass out quick all keep state

pass in quick on $int_if proto tcp from any to any port ssh flags S/SA keep state

block in log (all) all

devfs.rules:

[bastille_vnet=13]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path 'bpf*' unhide

These are added to sysctl.conf:

# Bastille:
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_member=0 # Packet filter on the member interface

What am I missing here? I have read the guide on bastillebsd.org and I can't see what's going wrong

 

[victort]

Can you ping the jails from the host?

 

[putney]

The jails are on the same subnet as the host re1 interface (192.168.3.100).

Perhaps try assigning this address to your re1bridge rather than re1.

ISTR reading that assigning addresses to bridge members has never worked reliably, though clearly it _has_ worked for most people most of the time. But it's certainly deprecated on 15.

 

[OpenFreeNet]

I'm using Bastille version 1.4.1.260315 and it writes a bad /usr/local/bastille/jails/<your-jail-name>/jail.conf configuration, in case of VNET jails.

I had to patch it manually. IIRC, there is a fix in recent version of Bastille.

This is an example of working configuration, for a mine jail called "streaming":

streaming {
  enforce_statfs = 2;
  devfs_ruleset = 4;
  exec.clean;
  exec.consolelog = /var/log/bastille/streaming_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = streaming;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/streaming/fstab;
  path = /usr/local/bastille/jails/streaming/root;
  securelevel = 2;
  osrelease = 14.3-RELEASE;


   vnet;
   vnet.interface = e0b_streaming;
   exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_streaming && ifconfig \${epair0%a}b up name e0b_streaming";
   exec.prestart += "ifconfig bridge0 addm e0a_streaming";
   exec.prestart += "ifconfig e0a_streaming description \"vnet0 host interface for Bastille jail streaming\"";
   exec.poststop += "ifconfig e0a_streaming destroy";
   allow.raw_sockets;
}

(edited) This is mine /etc/rc.conf

hostname="server1.local.lan"
keymap="it.kbd"

moused_nondefault_enable="NO"


sshd_enable="YES"
linux_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

# The interface where the server will receive the DHCP IP
# from modem/gateway, i.e. the external interface
ifconfig_bge3="DHCP"

# Create a bridge with the interfaces to use for DHCP connections.
# bridge0 is used both for DHCP connections and for connecting VNET jails.
gateway_enable="YES"
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0="addm bge0 addm bge1 addm bge2 addm tap0 addm tap1 addm tap3 addm tap2 up tap3 up"
ifconfig_bridge0_alias0="inet 192.168.2.1/24"
ifconfig_bge0="up"
ifconfig_bge1="up"
ifconfig_bge2="up"

dhcpd_ifaces="bridge0"

rcshutdown_timeout="900"
microcode_update_enable="YES"
sshd_dsa_enable="no"
sshd_ecdsa_enable="no"
sshd_ed25519_enable="YES"
sshd_rsa_enable="YES"
bastille_enable="YES"

unbound_enable="YES"

ntpd_enable="YES"
ntpd_sync_on_start="YES"

salt_master_enable="YES"
node_exporter_enable="yes"

ng_netflow_enable="YES"

ntpdate_enable="YES"
dhcpd_enable="YES"