Hello, all! I have an interesting setup that I'm hoping to get a little bit of help with. Before I start: please yell at me if some of my formatting is wonky-- I'm used to markdown, but not so much the styling that the FreeBSD Forums asks for, so if I've done something that needs to be different, I'm happy to learn. Okay: I have a network setup with a traffic flow that I want to resemble the following, from outermost to innermost:
Okay. So, I do have a custom-built kernel for IPFW (yes I know I can just load it at runtime, I chose not to for reasons; I'm not married to those reasons). Additions to the kernel are as follows:
In its current state of debugging, I've disabled some of the NAT configuration options that CBSD provides. Anyhow, here's my boot loader configuration:
And for good measure, here's my /etc/rc.conf:
I do have an IPFW ruleset at /etc/ipfw.rules
That being said, when I kick a Bhyve on (in this case, balance1a) I do get a modified ruleset because CBSD adds some rules.
(And yeah, I probably do need to fix some of the funkiness around rule 100 just for the sake of clarity.)
Okay, for CBSD, here's some settings I've transcoded from
The second entry for
Okay, I'm not particularly interested in IPv6 here. I do know for certain that the DHCPDISCOVER is making it out to the DHCP server, according to my OPNSense logs. I also do have logs indicating that the DHCP server is responding with a DHCPOFFER, and the MAC addresses matches the one in the code block I noted above.
I will note that I do have similar setups using similar hardware-- I do have a bhyve running in that same datacenter on a hypervisor pointed to an unmanaged switch, which is in turn uplinked to the same OPNSense box (just on a different interface). I also did (but don't currently) have boxes on the Mikrotik switch that did have bhyve instances on them-- but their physical interfaces weren't lagg'd. I've compared some of the above files between the box that's giving me trouble and the boxes that are working, and have tried to make them as similar as possible-- and this is as close as I've gotten at this point.
I've done some generic testing i.e. disabling the firewall to see if it's blocking things, trying to statically set the IP in the guest, etc, and I've not had luck. I'm happy to provide more information if it's needed, and I'm happy for any insight that y'all can provide. Thanks!
- Public-facing WAN on a bare-metal OPNSense instance; the OPNSense instance is also the DHCP server for everything under it (and this has been relatively successful), so while OPNSense serves multiple interfaces, each interface with a subnet, the one we're working with is on 10.0.9.0/24 (and the OPNSense instance / DHCP server is at 10.0.9.1)
- Mikrotik SFP+ switch with an uplink to an interface on the OPNSense box marked OPT1; the Mikrotik switch sits at 10.0.9.2
- Hypervisor host running FreeBSD 13.0-RELEASE with two physical interfaces dubbed bxe0 and bxe1 respectively, uplinked to the Mikrotik switch with two DAC lines (each individual line supports 10Gbps); neither of these interfaces has an IP assigned to it
- Cloned link aggregation interface on the hypervisor host dubbed lagg0 using the loadbalance protocol with members bxe0 and bxe1; the lagg interface doesn't have an IP address assigned to it
- Cloned bridge interface on the hypervisor host dubbed bridge0 with members lagg0 and tap1; the bridge interface has 10.0.9.33 successfully assigned to it through a static DHCP lease with a static ARP entry
- Automatically-generated tap1 interface on the hypervisor host, generated by CBSD on behalf of a bhyve.
- Guest-side virtual interface dubbed enp0s5
Okay. So, I do have a custom-built kernel for IPFW (yes I know I can just load it at runtime, I chose not to for reasons; I'm not married to those reasons). Additions to the kernel are as follows:
Code:
#
# IPFWKERNEL -- Generic kernel configuration file for FreeBSD/amd64 w/ IPFW support
# This needs to be located at /usr/src/sys/amd64/IPFWKERNEL
#
include GENERIC
ident IPFWKERNEL
# IPFW
options IPFIREWALL # required for IPFW
options IPFIREWALL_VERBOSE # optional; logging
options IPFIREWALL_VERBOSE_LIMIT # optional; don't get too many log entries
options IPDIVERT # needed for natd
In its current state of debugging, I've disabled some of the NAT configuration options that CBSD provides. Anyhow, here's my boot loader configuration:
Code:
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
if_lagg_load="YES"
kern.racct.enable=1
vmm_load="YES"
And for good measure, here's my /etc/rc.conf:
Code:
hostname="foo.bar.baz.com"
cloned_interfaces="lagg0 bridge0"
ifconfig_bridge0="addm lagg0 SYNCDHCP"
ifconfig_lagg0="laggproto loadbalance laggport bxe0 laggport bxe1"
ifconfig_bxe0="up"
ifconfig_bxe1="up"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"
cbsd_workdir="/usr/jails"
cbsdrsyncd_enable="YES"
cbsdrsyncd_flags="--config=/usr/jails/etc/rsyncd.conf"
cbsdd_enable="YES"
rcshutdown_timeout="900"
kld_list="vmm if_tuntap if_bridge nmdm"
I do have an IPFW ruleset at /etc/ipfw.rules
Code:
IPF="ipfw -q add"
ipfw -q -f flush
# Loopback
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag
# Stateful
$IPF 20010 check-state
$IPF 20020 allow tcp from any to any established
$IPF 20030 allow all from any to any out keep-state
$IPF 20040 allow icmp from any to any
# System Services
$IPF 20010 allow tcp from any to any dst-port 53 keep-state # DNS TCP
$IPF 20011 allow tcp from any to any dst-port 53 keep-state # DNS UDP
$IPF 20020 allow udp from any to any 67 in # DHCP SERVER IN
$IPF 20021 allow udp from any to any 67 out # DHCP SERVER OUT
$IPF 20022 allow udp from any to any 68 in # DHCP CLIENT IN
$IPF 20023 allow udp from any to any 68 out # DHCP CLIENT OUT
# User Services
$IPF 20100 allow tcp from any to any 2212 in # SSH ALT IN
$IPF 20101 allow tcp from any to any 2212 out # SSH ALT OUT
# Catch-All: Deny + Log
$IPF 65534 deny log all from any to any in
That being said, when I kick a Bhyve on (in this case, balance1a) I do get a modified ruleset because CBSD adds some rules.
ipfw list
thus yields:
Code:
00010 allow ip from any to any via lo0
00020 deny ip from any to 127.0.0.0/8
00030 deny ip from 127.0.0.0/8 to any
00040 deny tcp from any to any frag offset
00099 count ip from any to any out via tap1 // Setup by CBSD bhyve start: balance1a
00100 count ip from any to any in via tap1 // Setup by CBSD bhyve start: balance1a
20010 check-state :default
20010 allow tcp from any to any 53 keep-state :default
20011 allow tcp from any to any 53 keep-state :default
20020 allow tcp from any to any established
20020 allow udp from any to any 67 in
20021 allow udp from any to any 67 out
20022 allow udp from any to any 68 in
20023 allow udp from any to any 68 out
20030 allow ip from any to any out keep-state :default
20040 allow icmp from any to any
20100 allow tcp from any to any 2212 in
20101 allow tcp from any to any 2212 out
65534 deny log logamount 1 ip from any to any in
65535 deny ip from any to any
(And yeah, I probably do need to fix some of the funkiness around rule 100 just for the sake of clarity.)
Okay, for CBSD, here's some settings I've transcoded from
cbsd initenv-tui
- rcconf =
CBSD Enabled: YES
- nodename =
foo.bar.baz.com
- nodeip =
10.0.9.33
# note: the hypervisor host got this from DHCP, but CBSD got this from manual entry - nodeip6 =
0
- nodedescr =
0
- jnameserver =
10.0.9.1
- nodeippool =
10.0.9.0/24
- fbsdrepo =
1
- repo =
https://bsdstore.ru
- ipfw_enable =
1
- zfsfeat =
1
- jail_interface =
0
- parallel =
5
- stable =
0
- sqlreplica =
1
- statsd_bhyve_enable =
1
- statsd_jail_enable =
1
- statsd_hoster_enable =
1
- natcfg =
0 (not configured)
cbsd bconfig jname=balance1a
-- I've skipped some output because I can't imagine that it'd have anything to do with this)- ip4_addr =
10.0.9.129
# this is manually entered for CBSD, but it's also a static lease on the DHCP server - bhyvenic > nic1 =
vtnet
> nic_parent =bridge0
Code:
network:
version: 2
renderer: networkd
ethernets:
enp0s5:
dhcp4: yes
The second entry for
ip addr
reads as follows (I've included the enp0s5 entry but have excluded the lo entry):
Code:
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_code1 state UP group default qlen 1000
link/ether 00:a0:98:38:12:af brd ff:ff:ff:ff:ff:ff
inet6 fe80::2a0:98ff:fe38:12af/64 scope link
valid_lft forever preferred_lft forever
Okay, I'm not particularly interested in IPv6 here. I do know for certain that the DHCPDISCOVER is making it out to the DHCP server, according to my OPNSense logs. I also do have logs indicating that the DHCP server is responding with a DHCPOFFER, and the MAC addresses matches the one in the code block I noted above.
I will note that I do have similar setups using similar hardware-- I do have a bhyve running in that same datacenter on a hypervisor pointed to an unmanaged switch, which is in turn uplinked to the same OPNSense box (just on a different interface). I also did (but don't currently) have boxes on the Mikrotik switch that did have bhyve instances on them-- but their physical interfaces weren't lagg'd. I've compared some of the above files between the box that's giving me trouble and the boxes that are working, and have tried to make them as similar as possible-- and this is as close as I've gotten at this point.
I've done some generic testing i.e. disabling the firewall to see if it's blocking things, trying to statically set the IP in the guest, etc, and I've not had luck. I'm happy to provide more information if it's needed, and I'm happy for any insight that y'all can provide. Thanks!