Advice for Creating a "Command and Control" Server?

I have a problem that I think FreeBSD can solve for me. I have routers at various off-site locations that I need to be able to log into via SSH for monitoring, configuration, and maintenance. The problem is that none of the routers have static IP addresses, some of them are behind carrier NAT and firewalls, and some of the routers even have the same public IP address. I do have a FreeBSD server with a permanent public IP address and no ports blocked.
I'm thinking it would be easiest to create tunnels from the routers to the FreeBSD machine, but given the large number of options and the infinite configurations possible I felt it best to seek expert guidance first. The routers support EOIP, VPLS, OpenVPN, GRE, L2TP, PPP, and SSTP. As far as usage goes, the tunnels will be firewalled off from the users and carry only SSH management traffic. I'll be logging into the routers from the FreeBSD machine. I'd like to have a single interface/address on the FreeBSD machine and each router should have it's own IP address on the virtual network. I don't need layer 2 access.
What would you recommend and how would you recommend setting this up?


Profile disabled
In a qualitatively similar situation, only with not that much endpoints, I establish IPsec/IKEv2 (pre-shared key) tunnels between the peers. The important point here is, that the connection must be initiated by the peer with the dynamic or non-public IP (let's say the client) to the endpoint with the fixed well-known IP (the server), the other way around won't work. For IPsec management, I use security/strongswan. On the server side a route must be added for the connected clients. Once the VPN is established, the clients behave as if they were in the local network of the server, only the transmission rate may lack a bit, of course. While it is not exactly necessary, I like to establish the tunnels on private IPv4 alias addresses.

In case, this comes into question, we may discuss the setup in more detail.
i also have such a setup with openvpn
most of the clients are asus consumer routers with openvpn support
it works very well with some annoyances from the asus routers (they nat their lans thru the tunnel and i dont want that)
otherwise i find it easier to manage than ipsec
I think I came up with a good working solution using OpenVPN. I was having issues connecting at first, but it turned out that I had the encryption set to the wrong value in the router, it uses Blowfish128 by default, instead of the AES256 OpenVPN was expecting. I also had a rough time with the certificates, but that's mostly because OpenSSL documentation gives me a headache. I found an excellent tutorial for setting up an OpenVPN server on FreeBSD and eventually worked through it.

Since I'm not really going to be routing any traffic from the clients, my implementation differs from the tutorial in a few ways. I didn't need to add anything for routing in rc.conf. This is the condensed config file I used.

# Local address

# TCP/UDP port
port 1194

# TCP or UDP server?
proto tcp

# Layer 3 tunnel
dev tun

ca /usr/local/etc/openvpn/server/ca.crt
cert /usr/local/etc/openvpn/server/issued/server.crt
key /usr/local/etc/openvpn/server/private/server.key

# Diffie hellman parameters.
dh /usr/local/etc/openvpn/server/dh.pem

# Network topology
topology subnet

# Server mode and address range for subnet

# Maintain a record of client <-> virtual IP address
# associations in this file. 
ifconfig-pool-persist /root/ipp.txt

# Ping every 30 seconds and assume peer is disconnected if no response after 240 seconds
keepalive 30 240

# Cryptographic cipher
cipher AES-256-CBC

# The maximum number of concurrently connected
# clients we want to allow.
max-clients 1000

# Reduced privileges
user nobody
group nobody

# The persist options will try to avoid
# accessing certain resources on restart
# that may no longer be accessible because
# of the privilege downgrade.

# Output a short status file showing
# current connections, truncated
# and rewritten every minute.
status openvpn-status.log

# Log file verbosity.
verb 3

If it's of any interest, the routers I'm using are running RouterOS, this is how I configured them
import ca.crt password
import client.crt password
import client.key password
/interface ovpn-client
add certificate=ca.crt_0
add certificate=client.crt_0
set cipher=aes256
add name=ovpn-client1 connect-to= user=client
set disabled=no

Right now each client is getting an IP address from the server, none of the clients can talk to, or are aware of each other, the connection is routed instead of bridged so I'm not getting a bunch of unnecessary traffic, and most importantly I can log into the routers via SSH from the FreeBSD machine, all of which is pretty much ideal for my situation. Thank you very much.
for monitoring, configuration, and maintenance
This sounds like the classic requirements for standard (agent-based) orchestration + monitorig tools.

For orchestration you can have a look at the usual suspects: ansible, chef, puppet - all can work in push or pull direction. Ansible works without an agent on the remote side, pull functionality is (was?) achieved via git + cronjobs. Chef and puppet are using an agent that runs on the target host. All support connection over SSH or with TLS.

For monitoring I can highly recommend zabbix - it's agent also works in both directions (active vs passive checks) and natively supports authentication and TLS encryption. Especially its discovery funktions makes it ideal if you have a lot of (mostly) identical targets to monitor. Just create a template and every agent that connects to the server runs (on first connect and periodically afterwards) through pre-configured discovery jobs to catch configuration changes.


Staff member
it's agent also works in both directions (active vs passive checks) and natively supports authentication and TLS encryption.
Also don't forget about Zabbix proxy, which is quite useful when monitoring a remote location.