freebsd-update improvement idea

tux2bsd

Active Member

Reaction score: 63
Messages: 206

A simple method to improve freebsd-update with little effort for those with the coder-fu and necessary server side access.

e.g. current release = 13.0-RELEASE-p4

Code:
HTTP GET https://signal.update.freebsd.org/13.0-RELEASE-p4 if response is 200
then
  do nothing
else
  get back a 302 (or whatever, e.g. points to 13.1-RELEASE-p8 (because it's been a while in this fictional example...)
  do the normal freebsd-update fetch(/install), i.e. Inspecting system etc...

# server side burst rate limit N requests per hour, please yourself.

Then all of the crunching (Fetching/Inspecting...) only happens only if there is an update. Saving client side and server side resources markedly.
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

It can also handle multiple versions, an endpoint for 13.x and one for 14.x, rinse and repeat as versions move through time.

Over all the main goal to achieve is the reduction in time to process. freebsd-update dives into an intense self inspection prematurely, that should only happen AFTER a trivial preliminary test i.e. "is this self inspection necessary".
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

Or, it could simply be a helper script e.g. freebsd-update-signal but the server side component would need to be there to work...

This is probably the better approach. I chose the word signal as in traffic signal. Output and exit codes reflect outcome of freebsd-update-signal green no updates, orange point release update, red full version version.

Something like that...

freebsd-update-signal = green && echo yay
freebsd-update-signal = orange && freebsd-update fetch install (with env PAGER=cat)
freebsd-update-signal = red && freebsd-update cron && email_admin "wake up work to do"
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,685
Messages: 39,278

Note that /usr/sbin/freebsd-update is a (rather large) shell script, so you can easily look at the code to see what it does.
 

Zirias

Son of Beastie

Reaction score: 1,698
Messages: 2,868

This would be a misuse of HTTP. If you model a specific patch release as a resource, a redirect would mean this same resource is now moved, which obviously isn't the correct semantics.

You could model a release (without patch level) as a resource and do a conditional get, using the patchlevel as an Etag, for example. But why so complicated at all? I don't know what exactly freebsd-update(8) is doing right now, but fecthing e.g. a document like this
Code:
{
   "12.3": "p8",
   "12.4": "p6",
   "13.0": "p4"
}
would allow for a simple local check already. I assume something like this is done anyways?
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

Which benefits do you see that would be worth the costs?
Saving client side and server side resources markedly.
Client side: It takes too long to run freebsd-update, so long it's actually rather absurd. freebsd-update fetch ~1m20s (rpi3+, maybe that's halved on fast hardware) but it's still extremely high for each run, just to check! The method I'm proposing would be a single web request that would take less than ~100ms from most 1st world countries (if there's nothing to do - see above, method already explained).

Server side: The update servers would no longer serve the metadata each time, only when it is necessary. I don't work for FreeBSD so I'm not privy to how much benefit but it would definitely reduce load.

Of the two the significant benefit is seen on the client side.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,685
Messages: 39,278

Yeah, I looked at that too. But it has to have fetched updates first. And it's the fetch stage tux2bsd seems to have problems with.

You can speed things along a bit (especially if you have a bit of dodgy connection to the update servers) by using a caching proxy solution. That's particularly useful if you have to update/upgrade multiple machines. Besides only having one outgoing connection (only the proxy server has to have access to update.freebsd.org) the meta data and patch files themselves can be cached locally. This will speed up the downloads for all the other systems. But there's still quite a lot checking happening locally once the meta data has been downloaded. And to be honest that's a good thing, you do want to get all the right patch files that actually apply to your system. And if you only look at the version string there's a risk you're going to miss patches (especially if the previous update wasn't entirely completed).

Could this be improved, I'm sure it can. But I much rather have something running slow, steady and thorough then something that's fast and miss things.
 

kpedersen

Son of Beastie

Reaction score: 2,151
Messages: 2,999

It is my understanding that you need to fetch the metadata before your system can even work out whether you need to update anything.

Many times it detects the metadata has updated, so it fetches it, only to subsequently process it and realize that nothing that *I* have installed needs updating.

Yes, this can take a while tux2bsd is correct here. I am not quite getting >1min but often it is longer than fetching the actual packages I tend to install / update.

Potentially unless you need such up-to-date packages, the release or quarterly_[1,2,3,4] repos should have less of these metadata updates. My personal (perhaps less popular) solution is just make an offline mirror every 6 months. (only around ~98GB) ;)
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

This would be a misuse of HTTP...
That is your opinion. Making a small software utility decide what to do based on a controlled exchange of information is entirely achievable using HTTP(S), not every wheel needs reinventing.
And it's the fetch stage tux2bsd seems to have problems with.
The "fetch" is unnecessary (in my method). And it's actually the Inspection (of the fetch stage) that is most bizarre (if no updates do Inspection anyway... ???)
It is my understanding that you need to fetch the metadata before your system can even work out whether you need to update anything.
From above: freebsd-update dives into an intense self inspection prematurely, that should only happen AFTER a trivial preliminary test i.e. "is this self inspection necessary".
it could simply be a helper script e.g. freebsd-update-signal
This was the route I decided on. I will prototype it when I get time (that is a made up script name, it doesn't exist).
But I much rather have something running slow, steady and thorough then something that's fast and miss things.
It is a misunderstanding of computing to think that a quick result cannot be accurate. But with the (hypothetical) helper script you are free to a) not use it b) use it but perform freebsd-update by hand or however you always have - the onus is on the user.
 

Zirias

Son of Beastie

Reaction score: 1,698
Messages: 2,868

Some people, if they don't immediately understand what's wrong with their idea, why it's wrong and how it could be improved, do a sane thing: Reflect, research, maybe ask.

Some other people, not so much 🤡

On topic: If you really want to leverage HTTP itself for a version check, do it correctly (conditional GET with an Etag). Still, as described above, having a simple resource that contains the current patchlevels of all supported releases would serve the same purpose (a quick check with little network overhead), is simpler to implement and more flexible (e.g. also answers the question whether there's a new minor release, whether the current release is EOL, etc).
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,685
Messages: 39,278

Unfortunately not, because the (slow) fetch still has to occur.
It's not the fetching of the files from the server that's slow. It's the fact the Raspberry Pi is rather slow to read from an SD card.
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

I modified freebsd-update directly so that it performs a comparison and skips the "Inspection" if there are no new updates. It now takes between 0.7s and 2.1s, down from ~15s (this is on PC with SSD mentioned in the immediately previous post) the main variation being network induced (far away in NZ).

quick-freebsd-update-if-no-action.png
 

jbodenmann

Well-Known Member

Reaction score: 227
Messages: 433

Which benefits do you see that would be worth the costs?
I would still like to get back to my initial question here: What benefits do you see that make it worth the efforts? I fully understand that there is room for improvement, especially if looking only at a very specific metric such as "speed". However, usually updating the base system is a comparably rare occasion. It's not something that runs every 5 minutes periodically. I'd be really interested to know the situation which makes it necessary or worth addressing this.

Other than that, I think that SirDice already provided other solutions: Have a proxy/mirror nearby. That's a very easy, flexible & scalable approach.
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

initial question / Sirdice other solutions

Your initial question was answered. SirDice didn't provide any solution, certainly not one I was looking for but what he suggested may be a useful idea for someone else. SirDice comments regarding freebsd-update's code and the RPI SD card were appreciated and helpful.


It isn't about "speed", it is about reducing unnecessary inefficiency. It's that simple.

My underlying goal was to get freebsd-update fetch be more efficient when there are no updates. Originally I proposed a grandiose idea and that was just an idea (grandiose in that I never had any expectation anyone else would come to the party).

So.

What happened in reality: I did some experimenting with freebsd-update and found I could achieve most of my goal with no additional external aspect.

When there are no updates I've taken freebsd-update fetch from 15s down to 0.7s (that's on SSD, I'm yet to see what it does on the Raspberry Pi 3 (my prediction 1m20s down to ~5s).

I've reduced unnecessary inefficiency. I'm happy.
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

I'm yet to see what it does on the Raspberry Pi 3 (my prediction 1m20s down to ~5s).

Copied my change over... 1m20s down to 1.5s

A touch better than I expected.

Code:
root@freebsd-dns:/usr/sbin # date ; time freebsd-update.old fetch ; date
Thu Sep 30 21:31:23 NZDT 2021
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p4.
70.123u 12.270s 1:19.25 103.9%  17+171k 0+35io 0pf+0w
Thu Sep 30 21:32:42 NZDT 2021

#original above: freebsd-update.old
#modified below: freebsd-update

root@freebsd-dns:/usr/sbin # date ; time freebsd-update fetch ; date
Thu Sep 30 21:32:53 NZDT 2021
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
debug: Thu Sep 30 21:32:54 NZDT 2021
Already running 13.0-RELEASE-p4, no updates to perform.
0.421u 0.931s 0:01.51 89.4%     81+164k 0+8io 0pf+0w
Thu Sep 30 21:32:54 NZDT 2021
 

Zirias

Son of Beastie

Reaction score: 1,698
Messages: 2,868

Improving performance by (simple?) client-side changes definitely sounds like a good thing! Instead of showing your measurements, the actual change would be much more interesting though. Looking at the change, people could review it (and also double-check it doesn't break anything).

Of course, submitting it on bugzilla or phabricator would be best, cause that would have the best chance to receive meaningful reviews. If the change doesn't have unintended side-effects, it could be merged, so everyone can profit from it.
 
OP
T

tux2bsd

Active Member

Reaction score: 63
Messages: 206

Instead of showing your measurements, the actual change would be much more interesting though.
You need to work on your communication skills. If I did not take the time to make the change and show the measurements you would not be interested...

The change is simple.

I have yet to test the obvious case of when an update is available which I will do at some point with a fresh install.
 
Top