freebsd-update mirror

Tomse · Feb 17, 2017

Since I have quite a few FreeBSD installations on a totally closed network, I'm looking to
create a local mirror to update/upgrade these installations.

The handbook has this nice article
Build your own FreeBSD update server

This may or may not be what I need, but I have questions
as this seems to build the binaries rather than just get the binaries already available.
Am I wrong?
If I'm not wrong, are there alternatives, as I just need updates for 2 versions, my current
version + next version.. i.e. 10.2 latest + 10.3 latest. which will then later be replaced by 10.3 + 11 once 10.2 are no longer used.

Do anyone know if maintenance with a monthly update here would require 10's of minutes, hours or more/less (thinking manual work and not just waiting for the computer to download).

cheers

SirDice · Feb 17, 2017

Let me find it again, I have an Apache config that will proxy and cache the updates.

SirDice · Feb 17, 2017

Ok, you obviously need to change a few things but this works for me:

Code:

<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Order Deny,Allow
    Allow from All
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Order Allow,Deny
    Allow from All
  </Location>

  <IfModule cache_module>
    <IfModule disk_cache_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>

Then change the line in /etc/freebsd-update.conf:

Code:

ServerName fbsd-update.example.com

You'll also want to clean the cache regularly:

Code:

htcacheclean_cache="/var/cache/freebsd-update/"
htcacheclean_enable="YES"

On the network I maintain (around 30 servers) every server is set to update via the proxy. The proxy forwards to the official update servers and caches everything. So only the first machine that runs the update actually downloads them, the rest usually gets it from the cache.

I've set this up when our servers were all 9.1, they've since been upgraded to 9.3 and recently to 10.3. I didn't have to change anything or do anything special for that.

Tomse · Feb 20, 2017

How would this work on a closed network.

I connect the proxy server server to the internet, run freebsd-update from the server hosting the proxy, letting it update through the proxy.

What would happen when the proxy is back on the closed network, and clients differ from the proxy and some updates are not available?

cheers

Tabs · Feb 20, 2017

Maybe you could use freebsd-update fetch combined with --currently-running [I]release[/I] to populate the cache with updates for different versions?

Tomse · Feb 20, 2017

Tabs said:
Maybe you could use freebsd-update fetch combined with --currently-running [I]release[/I] to populate the cache with updates for different versions?

this sounds interesting I'll have a go at this tomorrow

SirDice · Feb 21, 2017

Tomse said:
How would this work on a closed network.

Ah, yeah, oops.

I'm not aware of any mirroring tools. But freebsd-update(8) is actually a large script, not a binary. Perusing through it I think it'll be fairly easy to write a script that simply fetches all the relevant files. The important functions appear to be fetch_metadata_index() and fetch_files(). I can easily imagine a script that fetches the metadata and pulls in all the relevant patch files.

Tomse · Feb 23, 2017

Tabs said:
Maybe you could use freebsd-update fetch combined with --currently-running [I]release[/I] to populate the cache with updates for different versions?

I didn't complete the test as of yet, but as I can see, it requires the updating server to be in sync with the clients (exact same patch level), it could be a good workaround, but in the long run might end badly.

SirDice said:
Ah, yeah, oops.

I'm not aware of any mirroring tools. But freebsd-update(8) is actually a large script, not a binary. Perusing through it I think it'll be fairly easy to write a script that simply fetches all the relevant files. The important functions appear to be fetch_metadata_index() and fetch_files(). I can easily imagine a script that fetches the metadata and pulls in all the relevant patch files.

if the freebsd-script gets this new function I think it would be able to do what I need, knowing my shell scripting skills I'm not being too optimistic about coding this myself

atleast I'll give it a try

scotia · Jul 2, 2018

SirDice said:

Ok, you obviously need to change a few things but this works for me:

Code:

<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Order Deny,Allow
    Allow from All
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Order Allow,Deny
    Allow from All
  </Location>

  <IfModule cache_module>
    <IfModule disk_cache_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>

Any chance that's cache_disk_module?

Code:

# apachectl -M | grep cache
 authn_socache_module (shared)
 file_cache_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 socache_shmcb_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)

Scott

SirDice · Jul 2, 2018

The configuration is a bit old, it's for Apache 2.2.

scotia · Jul 2, 2018

I thought that may be the case but it's worth recording for new arrivals to the thread (like me).

Scott

SirDice · Aug 8, 2018

Just remembered this thread, this is my current configuration, updated for Apache 2.4:

Code:

<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Require all granted
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Require all granted
  </Location>

  <IfModule cache_module>
    <IfModule cache_disk_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>

The previous Apache 2.2 configuration worked on 2.4 but didn't actually cache anything (as scotia correctly noted, the module is named differently). That should be fixed with this one.

rickvanderzwet · Sep 3, 2018

From NL I always seems to get the (quite slow) update5.freebsd.org update server based in US. As such I decided to setup a caching proxy with the above mentioned code at http://fbsd-update.rickvanderzwet.nl for public use. Feel free to try/test/use it.

ShelLuser · Sep 3, 2018

rickvanderzwet said:
From NL I always seems to get the (quite slow) update5.freebsd.org update server based in US.

Fellow Dutchie here, can't reproduce much issues myself.

rickvanderzwet said:
As such I decided to setup a caching proxy with the above mentioned code at http://fbsd-update.rickvanderzwet.nl for public use.

No offense intended but users are honestly better off to steer clear from that.

See, the problem is that none of us can get any guarantees that those packages you provide haven't been tampered with. It's simply a huge security risk to put your trust into unofficial sources like these, no matter how well intended they are.

xtaz · Sep 3, 2018

I live in the UK and I've actually added a DNS entry into my local DNS to repoint update5 to the IP address of update4 because it's so slow. I don't get why. I can ping/traceroute to it without any issue whatsoever, but when freebsd-update is trying to download anything from it it sits there and hangs for several minutes. When 11.2-RELEASE-p2 was released a couple of weeks ago I think it took around 5 minutes for it to download the 6 or 7 patch files. update4 in comparison is instant.

SirDice · Sep 3, 2018

ShelLuser said:
See, the problem is that none of us can get any guarantees that those packages you provide haven't been tampered with.

Update files are signed, so it would be fairly obvious if they've been modified.

Code:

# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <security-officer@FreeBSD.org> telling you to
# change it and explaining why.
KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5

balanga · Oct 23, 2022

SirDice said:
Just remembered this thread, this is my current configuration, updated for Apache 2.4:

Code:

<VirtualHost *:80> ServerAdmin info@example.com ServerName fbsd-update.example.com ProxyRequests Off ProxyPreserveHost Off <Proxy *> Require all granted </Proxy> ProxyPass / http://update.freebsd.org/ <Location /> ProxyPassReverse / Require all granted </Location> <IfModule cache_module> <IfModule cache_disk_module> CacheEnable disk / CacheRoot /var/cache/freebsd-update/ </IfModule> </IfModule> </VirtualHost>

The previous Apache 2.2 configuration worked on 2.4 but didn't actually cache anything (as scotia correctly noted, the module is named differently). That should be fixed with this one.

I'm looking into this topic and came across this thread which is four year old and wondered if anything needed updating...

SirDice · Oct 24, 2022

balanga said:
I'm looking into this topic and came across this thread which is four year old and wondered if anything needed updating...

Nope. Still works.

freebsd-update mirror

Tomse

SirDice

Administrator

SirDice

Administrator

Tomse

Tabs

Tomse

SirDice

Administrator

Tomse

scotia

SirDice

Administrator

scotia

SirDice

Administrator

rickvanderzwet

ShelLuser

xtaz

SirDice

Administrator

balanga

SirDice

Administrator