freebsd-update mirror

Tomse

Member

Thanks: 1
Messages: 36

#1
Since I have quite a few FreeBSD installations on a totally closed network, I'm looking to
create a local mirror to update/upgrade these installations.

The handbook has this nice article
Build your own FreeBSD update server

This may or may not be what I need, but I have questions
as this seems to build the binaries rather than just get the binaries already available.
Am I wrong?
If I'm not wrong, are there alternatives, as I just need updates for 2 versions, my current
version + next version.. i.e. 10.2 latest + 10.3 latest. which will then later be replaced by 10.3 + 11 once 10.2 are no longer used.

Do anyone know if maintenance with a monthly update here would require 10's of minutes, hours or more/less (thinking manual work and not just waiting for the computer to download).

cheers
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,335
Messages: 27,504

#2
Let me find it again, I have an Apache config that will proxy and cache the updates.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,335
Messages: 27,504

#3
Ok, you obviously need to change a few things but this works for me:
Code:
<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Order Deny,Allow
    Allow from All
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Order Allow,Deny
    Allow from All
  </Location>

  <IfModule cache_module>
    <IfModule disk_cache_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>
Then change the line in /etc/freebsd-update.conf:
Code:
ServerName fbsd-update.example.com
You'll also want to clean the cache regularly:
Code:
htcacheclean_cache="/var/cache/freebsd-update/"
htcacheclean_enable="YES"

On the network I maintain (around 30 servers) every server is set to update via the proxy. The proxy forwards to the official update servers and caches everything. So only the first machine that runs the update actually downloads them, the rest usually gets it from the cache.

I've set this up when our servers were all 9.1, they've since been upgraded to 9.3 and recently to 10.3. I didn't have to change anything or do anything special for that.
 
OP
OP
Tomse

Tomse

Member

Thanks: 1
Messages: 36

#4
How would this work on a closed network.

I connect the proxy server server to the internet, run freebsd-update from the server hosting the proxy, letting it update through the proxy.

What would happen when the proxy is back on the closed network, and clients differ from the proxy and some updates are not available?

cheers
 

Tabs

Member

Thanks: 20
Messages: 65

#5
Maybe you could use freebsd-update fetch combined with --currently-running release to populate the cache with updates for different versions?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,335
Messages: 27,504

#7
How would this work on a closed network.
Ah, yeah, oops.

I'm not aware of any mirroring tools. But freebsd-update(8) is actually a large script, not a binary. Perusing through it I think it'll be fairly easy to write a script that simply fetches all the relevant files. The important functions appear to be fetch_metadata_index() and fetch_files(). I can easily imagine a script that fetches the metadata and pulls in all the relevant patch files.
 
OP
OP
Tomse

Tomse

Member

Thanks: 1
Messages: 36

#8
Maybe you could use freebsd-update fetch combined with --currently-running release to populate the cache with updates for different versions?
I didn't complete the test as of yet, but as I can see, it requires the updating server to be in sync with the clients (exact same patch level), it could be a good workaround, but in the long run might end badly.


Ah, yeah, oops.

I'm not aware of any mirroring tools. But freebsd-update(8) is actually a large script, not a binary. Perusing through it I think it'll be fairly easy to write a script that simply fetches all the relevant files. The important functions appear to be fetch_metadata_index() and fetch_files(). I can easily imagine a script that fetches the metadata and pulls in all the relevant patch files.
if the freebsd-script gets this new function I think it would be able to do what I need, knowing my shell scripting skills I'm not being too optimistic about coding this myself :p
atleast I'll give it a try :)
 

scotia

Member

Thanks: 2
Messages: 39

#9
Ok, you obviously need to change a few things but this works for me:
Code:
<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Order Deny,Allow
    Allow from All
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Order Allow,Deny
    Allow from All
  </Location>

  <IfModule cache_module>
    <IfModule disk_cache_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>
Any chance that's cache_disk_module?

Code:
# apachectl -M | grep cache
 authn_socache_module (shared)
 file_cache_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 socache_shmcb_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
Scott
 

scotia

Member

Thanks: 2
Messages: 39

#11
I thought that may be the case but it's worth recording for new arrivals to the thread (like me).

Scott
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,335
Messages: 27,504

#12
Just remembered this thread, this is my current configuration, updated for Apache 2.4:
Code:
<VirtualHost *:80>
  ServerAdmin info@example.com
  ServerName fbsd-update.example.com

  ProxyRequests Off
  ProxyPreserveHost Off

  <Proxy *>
    Require all granted
  </Proxy>

  ProxyPass / http://update.freebsd.org/

  <Location />
    ProxyPassReverse /
    Require all granted
  </Location>

  <IfModule cache_module>
    <IfModule cache_disk_module>
      CacheEnable disk /
      CacheRoot /var/cache/freebsd-update/
    </IfModule>
  </IfModule>

</VirtualHost>
The previous Apache 2.2 configuration worked on 2.4 but didn't actually cache anything (as scotia correctly noted, the module is named differently). That should be fixed with this one.
 

ShelLuser

Son of Beastie

Thanks: 1,415
Messages: 3,073

#14
From NL I always seems to get the (quite slow) update5.freebsd.org update server based in US.
Fellow Dutchie here, can't reproduce much issues myself.

As such I decided to setup a caching proxy with the above mentioned code at http://fbsd-update.rickvanderzwet.nl for public use.
No offense intended but users are honestly better off to steer clear from that.

See, the problem is that none of us can get any guarantees that those packages you provide haven't been tampered with. It's simply a huge security risk to put your trust into unofficial sources like these, no matter how well intended they are.
 

xtaz

Well-Known Member

Thanks: 78
Messages: 341

#15
I live in the UK and I've actually added a DNS entry into my local DNS to repoint update5 to the IP address of update4 because it's so slow. I don't get why. I can ping/traceroute to it without any issue whatsoever, but when freebsd-update is trying to download anything from it it sits there and hangs for several minutes. When 11.2-RELEASE-p2 was released a couple of weeks ago I think it took around 5 minutes for it to download the 6 or 7 patch files. update4 in comparison is instant.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,335
Messages: 27,504

#16
See, the problem is that none of us can get any guarantees that those packages you provide haven't been tampered with.
Update files are signed, so it would be fairly obvious if they've been modified.

Code:
# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <security-officer@FreeBSD.org> telling you to
# change it and explaining why.
KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5
 
Top