I thought that converting a bunch of services from physical hardware to VPS would be fairly straightforward. I was sure that many people are doing this with FreeBSD. But I keep running into so many problems that I'm starting to wonder if it's possible.
The goal:
The goal:
- Spin up some small VPS servers, ideally doing binary installs and not installing /usr/src for speed of upgrading and minimizing storage space.
- Solved problem. Lots of vendors support VPS. FreeBSD supports doing binary installs.
- As these are small servers, use UFS to minimize memory footprint and support backups with dump.
- Create nullfs jails on them, using my own IP addresses, again doing binary installs.
- Initial plan was to use ezjail but that is not an option (see #4).
- Fell back to using bsdinstall jail but freebsd-update -b is unreliable at upgrading jails.
- iocage is not an option as it requires ZFS.
- No solution at present.
- Advertise those IP addresses via BGP.
- Solved problem. Install bird, do a little configuration, BGP routes get advertised.
- Be able to binary upgrade said jails.
- Major hurdle. For years, we've used ezjail to manage jails. But it's not really supported anymore and it now fails utterly in doing binary updates.
- Tried to roll my own but freebsd-update -b seems to miss files when updating jails.
- Extensive time using Google did not discover a fully working solution for doing binary-only upgrades of jails.
- No solution at present.
- Run VPNs between the VPS servers and the legacy hardware.
- Brought up OpenVPN which works fine at the link level.
- Run OSPF over the VPNs to deal with routing.
- Running into issues here. I'm testing on 11.1, with intent to upgrade to 11.2 once I solve issues #2 and #4 and have seen some posts suggesting that there may be some multicast issues with 11.1.
- I'm seeing OSPF Hello packets show up from the remote site to the VPS host but never see a response back from the VPS host as though the packets are being consumed by bird without actually doing anything with them.
- No solution at present.
- Back everything up regularly, ideally using dump.
- Solved problem as long as we are using UFS.