Just in case somebody else stumbles over this thread as I did when searching for some details on how to simplify/enhance my local caches, I'll add my current solution to this topic.
Firstly: There is no special package available for FreeBSD/pkg like e.g. apt-mirror on debian-based Linux systems. This is due to the fact, that a local caching mirror for even multiple pkg-based systems (this was what I was searching information for...), varying release versions and also freebsd-update is _really_ easy to set up with nginx.
Instead of multiple caches/repositories, I'm now using a single nginx installation within a jail to provide caches for all our local FreeBSD (pkg + freebsd-update) and TrueOS installations. The various caches/proxies are distinguished by their subdomain, e.g. freebsd-update.localdomain.lan; trueos-pkg.localdomain.lan etc.... It might be possible to distinguish by the location/path; but this way it would be an easy task to separate them if needed without the need to update configs on 100+ systems/jails.
The nginx.conf server entry for a cache looks like this:
Code:
server {
listen 80;
server_name freebsd-pkg.dmz.localdomain.lan freebsd-pkg.lan.localdomain.lan freebsd-pkg.localdomain.lan;
location / {
root /var/cache/packages/freebsd/;
try_files $uri @cache;
}
location @cache {
root /var/cache/packages/freebsd/;
proxy_store on;
proxy_pass https://pkg.freebsd.org;
proxy_cache_lock on;
proxy_cache_lock_timeout 20s;
proxy_cache_revalidate on;
proxy_cache_valid 200 301 302 24h;
proxy_cache_valid 404 10m;
}
}
The directory specified in both "root" entries has to be created and owned by the user nginx is running as (e.g. www).
To add another caching proxy, just add another server entry for another subdomain (or use different "location" directives) and create the cache directory. E.g. for freebsd-update:
Code:
server {
listen 80;
server_name freebsd-update.dmz.localdomain.lan freebsd-update.lan.localdomain.lan freebsd-update.localdomain.lan;
location / {
root /var/cache/freebsd-update/;
try_files $uri @cache;
}
location @cache {
root /var/cache/freebsd-update/;
proxy_store on;
proxy_pass https://update.freebsd.org;
proxy_http_version 1.1;
proxy_cache_lock on;
proxy_cache_lock_timeout 20s;
proxy_cache_revalidate on;
proxy_cache_valid 200 301 302 24h;
proxy_cache_valid 404 10m;
}
}
How it works:
If a host with this server in its pkg/freebsd-update configuration uses one of the specified server_names, nginx will look at the "root" directory set for that location if it already cached the requested files (e.g. packages). If true, it serves them directly from disk. If it isn't present it proxies the request to the URL specified in "proxy_pass", downloads the file to the local cache and (simultaneously) serves it to the local client. Every client asking for that file later will get it from the local cache.
To ensure the files are updated and/or expired, the proxy_cache_revalidate and proxy_cache_valid directives are set to check if there were changes on the origin server.
Using different locations might be the simpler solution if all caches will always be only on one system - as said I wanted to be able to easily migrate/switch single caches to another machine/jail if needed (e.g. to the same machine the build server is running on).
With the shown configuration this task would only require an update to the nameserver entries, not to all clients.
For further reading on fine-tuning the cache, the nginx blog and the admin-guide section contain some very useful articles:
https://www.nginx.com/blog/nginx-caching-guide/
https://www.nginx.com/resources/admin-guide/reverse-proxy/
Also have a look at the ngx_http_proxy_module:
https://nginx.org/en/docs/http/ngx_http_proxy_module.html
