Solved Howto save freebsd-update, portsnap, and pkg files to a fresh installation?

mjollnir

Daemon

Reaction score: 725
Messages: 1,167

Dear FreeBSD gurus!

Is it possible to save all those files in /var/db/{freebsd-update,pkg,portsnap} and /var/cache/pkg from a current installation (FreeBSD-12.1-RELEASE-p5) to a fresh installation of 12.1 (downloaded, with the official memstick.img, not selfmade from within /usr/src nor freebsd-update -r x.y upgrade), in order to save bandwidth and time to download all that stuff again?
I'm dialing in via WWAN module in my laptop, thus the data volume is limited. I do not want to download again what I already have.

I know with the pkg's it should be possible, but how about freebsd-update(8) and portsnap(8) files?
I want to repartition my disk (+ a "Rapid Start" partition) so I think I have to re-install, right? The current disk layout is "full" and does not allow for moving partitions:
Code:
=>       40  500118112  ada0  GPT  (238G)
40     409600     1  efi  (200M)
409640 1024 2 freebsd-boot (512K)
410664 984 - free - (492K)
411648 16777216 3 freebsd-swap (8.0G)
17188864 482928640 4 freebsd-zfs (230G)
500117504 648 - free - (324K)
Or does ZFS allow for shrinking it's size? It's utilization is 40%.

Thx in advance for any hint.
 
Last edited:
OP
mjollnir

mjollnir

Daemon

Reaction score: 725
Messages: 1,167

SOLVED
I apologize distracting your attention on an issue that I could easily solve myself:

It's not worth it:
Code:
# du -schx /var/db/* | sort --human
[...]
47M    freebsd-update
84M    pkg
207M portsnap
237M clamav
581M total
# du -schx /var/cache/* | sort --human
[...]
1,0G    /var/cache/pkg
5,0G    /var/cache/ccache
6,0G total
and since I remember I succeded once with saving the packages in /var/cache/pkg, I can now reinstall and during that add a "Rapid Start" partition. Hope suspend/resume will work then.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,282
Messages: 33,825

If you have a lot of machines to run freebsd-update(8) on I can recommend setting up a caching proxy specifically for this. Not only will this force all the updates through a single machine (so not all machines need to have access to the outside world), it will also get the update files from the cache if they've been downloaded by a previously run freebsd-update(8). If you have 20 machines then only the first one will actually download the files, the rest will get it from the proxy's cache.
 
OP
mjollnir

mjollnir

Daemon

Reaction score: 725
Messages: 1,167

If you have a lot of machines to run freebsd-update(8) on I can recommend setting up a caching proxy specifically for this. [...]
Yes, thx.
Not a lot, but even for two machines this makes sense, esp. since I'm dialing in via WWAN. A www/squid or such, and if I want some special setup on my machines, I could use the FreeBSD Update Server (local link).
The 1st solution should be simpler and also serve for data like updates for security/clamav.
I'm going to save /boot/loader.conf, {,/usr/local}/etc (and weave in only selected changes I made on the fresh intallation), /var/db/{clamav,freebsd-update,pkg,portsnap} and /var/cache/{pkg,ccache} in addition to my personal data. If that fails I'm not worried, since the amount of data to download again is not overly much. Then among the 1st tasks on the fresh installation is to install a caching proxy.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,282
Messages: 33,825

Not a lot, but even for two machines this makes sense, esp. since I'm dialing in via WWAN.
For my home network I configured my firewall/router to do this. It wasn't doing much and I had plenty of space to spare.

Simple nginx.conf:
Code:
http {
    include       mime.types;
    default_type  application/octet-stream;

    proxy_cache_key "$scheme$request_method$host$request_uri";
    proxy_cache_path /var/cache/fbsd-update levels=1:2 keys_zone=fbsdupdate_cache:10m
                      max_size=5G inactive=14d use_temp_path=off;

    sendfile        on;
    keepalive_timeout  65;

    server {
      listen 192.168.x.1:80;
      server_name fbsd-update.example.com;

      root /var/cache/fbsd-update;

      access_log /var/log/nginx/proxy-access.log;

      location / {
        proxy_cache fbsdupdate_cache;
        proxy_cache_lock on;
        proxy_buffering on;
        proxy_http_version 1.1;
        proxy_cache_revalidate  on;
        proxy_cache_valid      200  7d;
        expires max;
        add_header X-Proxy-Cache $upstream_cache_status;

        proxy_pass http://update.freebsd.org;
      }
    }
}
That fbsd-update.example.com resolves internally to the 192.168.x.1 address. I then change the ServerName in freebsd-update.conf to fbsd-update.example.com. This pushes everything through this caching proxy. The fist system I update is a bit slow, as that triggers the downloads. Consecutive updates on the other machines get their files mostly from this cache and only download from upstream if there's something missing.
 
OP
mjollnir

mjollnir

Daemon

Reaction score: 725
Messages: 1,167

For my home network I configured my firewall/router to do this. It wasn't doing much and I had plenty of space to spare.
Simple nginx.conf:
Code:
[...]
        proxy_http_version 1.1;
[...]
  • So with www/nginx I have a web server + caching proxy, and www/squid is not fully HTTP 1.1 compliant, or is the info for the port outdated?
  • There are many utilities in the ports to process www/squid log files to produce nice statistics and graphics, and to configure www/apache, will they understand www/nginx config & logs, too?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,282
Messages: 33,825

I already had nginx running for my letsencrypt certificates. So it was easier for me to just add another virtual host to it, without having to install additional software. For a client I set up a caching proxy with Apache, because I already had Apache installed for other things. It really doesn't matter which one you use, they can all be configured to do pretty much the same thing.
 
OP
mjollnir

mjollnir

Daemon

Reaction score: 725
Messages: 1,167

Just for the record, in case others have the same issue (save data volume on a dial-in internet connection):
  • pkg(8) is fine with re-using a saved /var/cache/pkg
    It did not help much, though, because many of the packages were rebuild on the upstream server while I was re-installing my machine... :/
  • I do not know how freebsd-update(8) and portsnap(8) liked their saved data files; in both cases the absolute volume to save is rather low, so I did not further investigate this.
  • I did not save ccache(1)'s cache, since it's data is easily reproducible.
    This would have been the one to save the greatest amount of time, though.
  • Last not least, I succeeded with suspend/resume to the new Intel Fast Flash/Intel Rapid Start (iFFS/IRST) partition! :cool:
    A How-To follows.
 
Top