freebsd-update keeps failing

B

balanga

I've been having endless problems trying to run freebsd-update on various sytems which all seem to end in failure. I'm currently attempting to upgrde 12.2 to 13.1 and have already fetched 1000's of patches. If I run the update again does it start from scratch or continue where it left off?

Can anyone suggest a way to get round my problem? Maybe manually select an update server?..
 
You don't mention how they fail. I have problems when I use ksh, with my environment, path, and aliases. So I always completely sanitise the environment before running the update:
Code: 
/usr/bin/env - /bin/sh
freebsd-update fetch
freebsd-update install
exit
 
  • Like
Reactions: mer
I've not experienced issues, but with what gpw928 says about environment and shell is a good idea. I've run into issues where bash was assumed and using tcsh caused wierd issues. That's one thing I dislike about most Linux distros: they have all kinds of aliases variables set by default that annoy me.
 
gpw928 said:
You don't mention how they fail. I have problems when I use ksh, with my environment, path, and aliases. So I always completely sanitise the environment before running the update:
Code: 
/usr/bin/env - /bin/sh
freebsd-update fetch
freebsd-update install
exit
Click to expand...
I do exactly that with a shell script.

Bash: 
#!/usr/bin/env bash

if [[ "${UID}" != "0" ]] ; then echo "UID != 0" ; exit ; fi

printexec() {

    printf "\e[91;1m * ${*}\e[0m\n"
    "${@}"

}

#if [[ $(date +%u) -eq 3 ]] || [[ $(date +%u) -eq 7 ]] ; then

    printexec freebsd-update fetch install
    printexec pkg update
    printexec pkg upgrade
    printexec portsnap fetch update
    printf "\e[91;1m * Done\e[0m\n"
    exit 0

#fi

Works like a charm every time.
 
Have you tried to verify a failed hash by hand? A failed hash is "computed hash does not match what someone else says the hash should be".
I'm not sure where the one to compare against is stored (maybe locally on the system, maybe stored on a remote site).
It's also possible that the stored value should have been updated but wasn't for some reason; that would be a false negative.
Have you tried running just the freebsd-update fetch command by hand in a clean environment, just to eliminate everything else?
It looks like freebsd-update has an "-s" which should let you pick a specific server to use, maybe try that when running it by hand.
 
SirDice said:
Files are cached in /var/db/freebsd-update.
Click to expand...

Are you saying that if I just keep re-running the update I will eventually get all the files I need? For example when fetching 801 files and I get an 'incorrect hash' with the last file, have the previous 800 been retrieved correctly?

In /var/db/freebsd-update/files/ I have almost 40,000 files going back to 2019. Is there any way of identifying what they relate to and would I be able to copy them to a 'freebsd update server' if/when I manage to set one up?
 
balanga said:
For example when fetching 801 files and I get an 'incorrect hash' with the last file, have the previous 800 been retrieved correctly?
Click to expand...
It should, yes.

balanga said:
In /var/db/freebsd-update/files/ I have almost 40,000 files going back to 2019. Is there any way of identifying what they relate to and would I be able to copy them to a 'freebsd update server' if/when I manage to set one up?
Click to expand...
No idea. You don't need to set up your own update server. Just set up a caching proxy for this. It's all 'standard' HTTP traffic and the files are easily cached with a caching proxy server. So the next server you run freebsd-update(8) on will get the files from the proxy's cache. Which will be much faster than constantly trying to fetch these files from update.freebsd.org.
 
mer said:
Have you tried to verify a failed hash by hand? A failed hash is "computed hash does not match what someone else says the hash should be".
I'm not sure where the one to compare against is stored (maybe locally on the system, maybe stored on a remote site).
It's also possible that the stored value should have been updated but wasn't for some reason; that would be a false negative.
Have you tried running just the freebsd-update fetch command by hand in a clean environment, just to eliminate everything else?
It looks like freebsd-update has an "-s" which should let you pick a specific server to use, maybe try that when running it by hand.
Click to expand...
The only time I see a filename shown is when an 'incorrect hash' is reported. I suspect that in such a case that file is not stored on the system.

If I want to use the '-s' option, how do I find which servers are available?

I think I will need to deploy script() to keep track of my attempts.

When running 'fetch' as a separate step, does that retrieve all the required files? And if there has been a previous fail, does it make use of previously retrieved files?
 
Funny, I updated the 13-RELEASE kernel (in a VirtualBox VM) to 13.1-RELEASE just following the Handbook... I think about 20k patches got downloaded, that part took up about half an hour... For me, it boiled down to keeping track of the steps:
Code: 
# freebsd-update -r 13.1-RELEASE upgrade
...
(Stuff happens. In my case,  I think I first had patches downloaded, and then I had to edit /etc/groups because somehow the update would leave me out of the video group... :/ )
...
# freebsd-update install
# reboot
(after reboot)
Code: 
# freebsd-update install
# uname -a
(should reflect the update)

That's it. Sometimes, keeping it simple yields the best results.
 
  • Like
Reactions: mer
balanga To the best of my knowledge, freebsd-update fetch should pick up where it left off. If it needs to get 100 files, fails at file 51, when you rerun it I think it starts at file 51. It may do a quick check of the others (maybe a hash check) but I think it picks it up. It's been a while since I've run into a failure so I'm going by memory.
 
VladiBG said:
Is the server that you are updating is physical server or virtual machine? I'm not asking from where you are updating.
Click to expand...
Why would the difference between physical and virtual matter? If you do steps correctly (as Handbook shows you), that physical/virtual difference really shouldn't matter. Not for update server, and not for machine being updated... 😕

Edit: FWIW, that update from my earlier post - that was just a quick chore I did last night...
 
astyle said:
Funny, I updated the 13-RELEASE kernel (in a VirtualBox VM) to 13.1-RELEASE just following the Handbook... I think about 20k patches got downloaded, that part took up about half an hour... For me, it boiled down to keeping track of the steps:
Code: 
# freebsd-update -r 13.1-RELEASE upgrade
...
(Stuff happens. In my case,  I think I first had patches downloaded, and then I had to edit /etc/groups because somehow the update would leave me out of the video group... :/ )
...
# freebsd-update install
# reboot
(after reboot)
Code: 
# freebsd-update install
# uname -a
(should reflect the update)

That's it. Sometimes, keeping it simple yields the best results.
Click to expand...
I know the theory and it has worked without problems on numerous occasions in the past but I seemed to have had a lot of issues with upgrading to 13.1-RELEASE. I guess I should assume that I simply have a unreliable connection.

As I understand it all the fetches are cumulative and I just need to run the update over and over until I get all the required files.
 
astyle said:
Why would the difference between physical and virtual matter? If you do steps correctly (as Handbook shows you), that physical/virtual difference really shouldn't matter. Not for update server, and not for machine being updated... 😕

Edit: FWIW, that update from my earlier post - that was just a quick chore I did last night...
Click to expand...
In some virtual machines the hardware based checksum(Offloading) of the RX/TX for the NIC need to be disabled so it can be done on software level.
The problem with the receiving a corrupted (damaged) file which is catch by the checksum verification of freebsd-update means that he has some sort of connection issue. I would reset the TCP counters with netstat -sz then perform a freebsd-update and after receiving again an incorrect hash error will check the TCP statistic with netstat -ss -p TCP
 
balanga said:
I know the theory and it has worked without problems on numerous occasions in the past but I seemed to have had a lot of issues with upgrading to 13.1-RELEASE. I guess I should assume that I simply have a unreliable connection.

As I understand it all the fetches are cumulative and I just need to run the update over and over until I get all the required files.
Click to expand...
I think the command # freebsd-update -r 13.1-RELEASE upgrade will actually let you know when everything is done and ready for # freebsd-update install... The important part is to keep re-running the first command until it finishes successfully.
 
  • Like
Reactions: mer
VladiBG said:
In some virtual machines the hardware based checksum(Offloading) of the RX/TX for the NIC need to be disabled so it can be done on software level.
The problem with the receiving a corrupted (damaged) file which is catch by the checksum verification of freebsd-update means that he has some sort of connection issue. I would reset the TCP counters with netstat -sz then perform a freebsd-update and after receiving again an incorrect hash error will check the TCP statistic with netstat -ss -p TCP
Click to expand...
Last time virtual NICs had that kind of problem was back in 2010s, when virtualization was a newcomer to the market. Since then, codebases have improved, so ending up with a corrupted download is less of an issue. Or was there a recent event where an up-to-date VM ended up with a corrupted download (preferably involving freebsd-update)?
 
balanga said:
Sometimes it just says fail, sometimes I get 'file' has incorrect hash when the last of say 1054 files have been fetched.
Click to expand...

I have the same issue on every update. Cause is a Sophos UTM firewall which does transparent http-proxying. I solved this by ssh-tunneling http-traffic to a different squid-proxy on the internet / outside the LAN. Works flawlessly (and much *much* more faster for subsequent hosts).

I also tried to use a squid proxy inside the LAN, but this only cached the 'incorrect hash' and I had to clean up the cache manually all the time.

Besides that, I can confirm that using a squid proxy can speed up updates by factor 100x, especially if you have to update multiple machines on the same LAN.
 
xinqu said:
I have the same issue on every update. Cause is a Sophos UTM firewall which does transparent http-proxying. I solved this by ssh-tunneling http-traffic to a different squid-proxy on the internet / outside the LAN. Works flawlessly (and much *much* more faster for subsequent hosts).

I also tried to use a squid proxy inside the LAN, but this only cached the 'incorrect hash' and I had to clean up the cache manually all the time.

Besides that, I can confirm that using a squid proxy can speed up updates by factor 100x, especially if you have to update multiple machines on the same LAN.
Click to expand...
Wouldn't that mean that you gotta have a ton of storage space for the proxy cache?
 
freebsd-update relies on the proxy server string in the environment to seed the random selection of mirrors. If this isn't set consistently across installations, it leads to a low hit rate in the cache. With an intercepting proxy (which is what people usually mean when they say transparent proxy) freebsd-update makes no attempt to use consistent URIs. In that case it may be possible to go directly to the proxy by setting the environment variable.

One other thing is that freebsd-update relies heavily on HTTP pipelining. If a proxy can't handle that efficiently it can make the "fetch" stage very slow.
 
You must log in or register to reply here.
Back
Top