OP
- Thread Starter
- #26
Getting back at it - I'm in process trying to install BIND on both servers.
- SR-IOV continues to behave exactly the way I'd want when combined with vnet jails.
- 11 VFs created on ixl2 (the 1st port of my X710-T2L) and 6 VFs created on ixl3 (the 2nd port). ixl0 & 1 are the onboard ports on my Supermicro X11DPH-i.
- It's not well-documented, but it is possible to create and use a separate .conf file for each interface that's getting configured with VFs: I have ixl2.conf and ixl3.conf located in /etc/iov/.
- The syntax to use in
/etc/rc.confis where I had a problem... couldn't figure out how to get both files to load. Eventually I stumbled on a post somewhere (and have since lost the link) that said to simply list the files in a single line with a space separator:Code:iovctl_files="/etc/iov/ixl2.conf /etc/iov/ixl3.conf" - Some extra fun you can have is assigning whatever MAC address you want to VF interfaces.
- Just specify
allow-set-mac : true;at the top of the iov .conf file, and within each VF definition specify the MAC address you want:mac-addr : "XX:XX:XX:XX:XX:XX"; - According to the IANA, addresses that start in the range 00:00:00 to 00:00:FF are reserved (what for? I don't know).
- So, I set up my VF MAC addresses using the various jersey numbers my kid wore during his youth hockey career... 00:00:53:24:02:XX. Doesn't seem to cause any problems with my networking gear, so whatever that range is reserved for clearly isn't important (and doesn't exist/conflict).
- Just specify
- 11 VFs created on ixl2 (the 1st port of my X710-T2L) and 6 VFs created on ixl3 (the 2nd port). ixl0 & 1 are the onboard ports on my Supermicro X11DPH-i.
- BIND installed on the jail running on the primary server without issue.
- In order to get jails working on the secondary server (running on consumer-grade hardware), I needed to use a bridge interface. SR-IOV is not an option on the Z270 motherboard, nor does the X540-T2 NIC support it.
- This machine has been running a pair of bhyve VMs with my old Windows Servers (and very reliably, I might add), so I thought I'd keep things simple and just use one bridge for bhyve and jails. Once the Windows VMs are decommissioned I won't need to do anything except shut them down and archive (or delete) them.
- Implementation for this is pretty straightforward - no gotchas lurking. The bridge interface that's created in the background when running the
vm switch createcommand doesn't seem like the best choice, since I don't think it'll persist after I get rid of the VMs and destroy the vm switch. It is possible to set up a bridge first and then force the vm switch to use it. Steps I followed:- Shut down both Windows VMs
vm switch destroy public(I didn't see a need to get creative with the switch name)- Set up the bridge in /etc/rc.conf, adding the ix0 interface and assigning the host IP to the bridge instead of the ethernet port.
- update the host to 14.3-RELEASE-p5 since I'm here...
- reboot and check
ifconfigto see thatbridge0is set up properly. Verify connectivity to the outside world. - Re-create the vm-bhyve switch using the existing bridge:
vm switch create -t manual -b bridge0 public- I gave it the same name (public) so I wouldn't need to modify the VM network configs at all.
- Start both VMs back up and verify connectivity
- Worked like a charm, so I moved on to trying to get BIND installed on a jail on the secondary server.
- My jail.conf on the secondary server has all the appropriate
exec.prestart/start/stop/poststopdirectives for dynamically creating, attaching, detaching, and destroying epairs for the jails when they're started/stopped. Works great. If anyone cares, here's a privacy-redacted version:# STARTUP/LOGGING
exec.prestart = “logger Starting jail ${name}…”;
exec.prestart += ”/sbin/ifconfig ${epair} create up”;
exec.prestart += ”/sbin/ifconfig ${epair}a up descr jail:${name}”;
exec.prestart += ”/sbin/ifconfig ${bridge} addm ${epair}a up”;
exec.start = “/sbin/ifconfig ${epair}b ${ip} up”;
exec.start += “/sbin/route add default ${gateway}”;
exec.start += “/bin/sh /etc/rc”;
exec.poststart = “logger Started jail ${name}.”;
exec.prestop = “logger Stopping jail ${name}…”;
exec.stop = “/bin/sh /etc/rc.shutdown”;
exec.poststop = “logger Stopped jail ${name}.”;
exec.poststop += “/sbin/ifconfig ${bridge} delete ${epair}a”;
exec.poststop += “/sbin/ifconfig ${epair}a destroy”;
exec.timeout = 90;
exec.consolelog = “/var/log/jail_console_${name}.log”;
# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = “5”;
# NAME/PATH
path = “/usr/local/jails/containers/${name}”;
host.hostname = “${name}.somedomain.com”
# NETWORKS/INTERFACES
$ip = “10.0.0.${id}/24”;
$gateway = “10.0.0.1”;
$bridge = “bridge0”;
$epair = “epair${id}”;
jail10 {
$id = “10”;
vnet;
vnet.interface = “${epair}b”;
}
jail11 {
$id = “11”;
vnet;
vnet.interface = “${epair}b”;
}
jail12 {
$id = “12”;
vnet;
vnet.interface = “${epair}b”;
}
jail13 {
$id = “13”;
vnet;
vnet.interface = “${epair}b”;
} - Last night's hiccup: I really have no idea what I'm doing when it comes to the pf firewall.
- Unlike on my primary server, where
pkg -j jailname install bind920worked without a hitch, I got some sort of permission denied error when using pkg install against a jail on the secondary machine. - The only difference is the network config - I'd wondered before if pf would behave the same on a machine using bridged networking for the jails, and I'm fairly certain the answer is no.
- This morning I actually went and read the PACKET FILTERING section of the if_bridge(4) manpage (should have done that before), and I think it confirms my suspicions.
- I don't think the per-jail pf configuration I'm using on the primary server will work... the manpage says that all inbound packets are filtered on the originating interface (ix0 in my case), while outbound packets are filtered on the "appropriate interfaces." My guess is that it means outbound packets are filtered on
epairbinterfaces...
- I'll have to toy with this a bit to see what's actually happening and adjust my firewalling strategy. I'm not inclined to simply shut off pf so I can install BIND - I'd rather have the pf setup procedure completely ironed out for a simple service that uses one UDP port and one TCP management port. When I go to set up more involved jails I really don't want to be flailing around wondering why things are behaving poorly.
- Unlike on my primary server, where
- Assuming I'm right and it's just my lack of understanding about how filtering works with bridge interfaces, and assuming I learn enough to move on, I'm fairly confident that I'll get BIND up and running in the next day or two.
- Simple setup: a single forward lookup zone and a few reverse lookup zones, all of which are private (so I don't need to mess around signing things for DNSSEC). These BIND instances will also act as resolvers - not best practice for networks with large loads, but mine is small and has comparatively light traffic. Should be fine. Filter lists will wait a bit, because...
- I hadn't planned on it originally, but it looks as though I'll be implementing a DHCP jail. I run a Unifi UDM as my router, and I've been using the built-in DHCP service. Unfortunately Unifi doesn't seem to offer any support for RFC2136 DDNS updates from their DHCP service, which is an (maybe not surprising) unpleasant discovery. Seriously... this stuff has been around for ages. There are some documented attempts I've seen to install nsupdate on a Unifi device, or even add a middleware layer to make the UDM think it's updating a cloud DDNS provider (which it supports). I'm not interested. If Unifi DHCP won't natively interface with a standards-compliant DNS server, I'll just roll my own.
- So, as soon as I have my static zones set up and working properly in BIND, I'll make the tweaks necessary to support DDNS updates and go ahead with a DHCP jail. I'm thinking Kea, since it's the currently-supported offering from ISC. The old ISC dhcpd 4.4 offering was EOL-ed in 2022... Seems like the FreeBSD handbook should be updated to reflect this. I'm sure it works fine, but EOL means minimal (or no) security fixes, and that's not ok with me.
(or less energy use in-general with less machines)