how to recover from failed 11.4-RELEASE to 12.0 upgrade?

D

Dave_Berg

I've got a server about 50 miles away that failed to upgrade. I wanted to
end up at 12.3-RELEASE, but I figured going to 12.0 first would be a good
idea. I ran the usual:

Code: 
sudo /usr/sbin/freebsd-update -r 12.0-RELEASE upgrade
dave@mail % sudo /usr/sbin/freebsd-update -r 12.0-RELEASE upgrade
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 11.4-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.

Everything seems to go well, but right after merging syslog.conf, there's
one odd thing:

Code: 
 local0.*           /var/log/haproxy.log
 local1.*           /var/log/custom.log
Does this look reasonable (y/n)? y
look: INDEX-NEW: No such file or directory
look: INDEX-NEW: No such file or directory
look: INDEX-NEW: No such file or directory

this goes on for a bit, then:
Code: 
comm: INDEX-NEW: No such file or directory
rm: modifiedfiles: No such file or directory
rm: INDEX-PRESENT: No such file or directory

It all seems to go well after that, until:
Code: 
/boot/kernel/fade_saver.ko
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

I rebooted, and ran:
Code: 
dave@mail % sudo /usr/sbin/freebsd-update install
Installing updates...

and then I looked away for a bit. When I came back, the screen is full of:
Code: 
chflags: ///usr/src/tests/sys/cddl/zfs/include/libgnop.kshlib: No such file or directory
chflags: ///usr/src/tests/sys/cddl/zfs/include/libremote.kshlib: No such
file or directory

Odd, but it's in /usr/src, I don't care about that now. chflags goes on a
long time, eventually this pops up:

Code: 
chflags: ///var/db/etcupdate/current/etc/syslog.d/ppp.conf: No such file or directory
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libedit.so.7: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: ld-elf.so.1: /lib/libmd.so.6: invalid file format
/lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: ld-elf.so.1: /lib/libmd.so.6: invalid file format/lib/libmd.so.6: invalid file format

 done.
2:38.36
dave@mail[~] % su -
su: pam_start: System error
dave@mail[~] % sudo reboot
sudo: unable to initialize PAM: Exec format error

I looked in /lib, and sure enough, there's mostly libs from 11.4 and some from
12.0, and some that are just empty files:
Code: 
dave@mail[~] % man freebsd-update
ld-elf.so.1: /lib/libedit.so.7: invalid file format
dave@mail[~] % ll /lib
total 7081
drwxr-xr-x  2 root  wheel    13B Aug 28 15:54 casper/
drwxr-xr-x  2 root  wheel    18B May 26 14:50 geom/
-r--r--r--  1 root  wheel    23K Aug 28 15:54 lib80211.so.1
-r--r--r--  1 root  wheel    51K Aug 28 15:54 libalias.so.7
-r--r--r--  1 root  wheel   5.9K May 26 14:50 libalias_cuseeme.so
-r--r--r--  1 root  wheel   5.2K May 26 14:50 libalias_dummy.so
-r--r--r--  1 root  wheel    12K May 26 14:50 libalias_ftp.so
-r--r--r--  1 root  wheel   9.2K May 26 14:50 libalias_irc.so
-r--r--r--  1 root  wheel   7.9K May 26 14:50 libalias_nbt.so
-r--r--r--  1 root  wheel   8.3K May 26 14:50 libalias_pptp.so
-r--r--r--  1 root  wheel   5.9K May 26 14:50 libalias_skinny.so
-r--r--r--  1 root  wheel   9.4K May 26 14:50 libalias_smedia.so
-r--r--r--  1 root  wheel    15K Aug 28 15:54 libavl.so.2
-r--r--r--  1 root  wheel    35K Aug 28 15:54 libbe.so.1
-r--r--r--  1 root  wheel    19K Aug 28 15:54 libbegemot.so.4
-r--r--r--  1 root  wheel   178K Aug 28 15:54 libbsdxml.so.4
-r--r--r--  1 root  wheel   1.9M Aug 28 15:54 libc.so.7
-r--r--r--  1 root  wheel   183K Apr  6  2019 libcam.so.6
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libcam.so.7
-r--r--r--  1 root  wheel    22K Jul 20 15:33 libcasper.so.0
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libcasper.so.1
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libcrypt.so.5
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libcrypto.so.111
-r--r--r--  1 root  wheel   1.9M Apr  6  2019 libcrypto.so.7
-r--r--r--  1 root  wheel   2.4M Aug 26 16:19 libcrypto.so.8
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libctf.so.2
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libcxxrt.so.1
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libdevstat.so.7
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libdtrace.so.2
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libedit.so.7
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libelf.so.2
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libgcc_s.so.1
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libgeom.so.5
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libibverbs.so.1
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libipsec.so.4
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libipt.so.0
-r--r--r--  1 root  wheel   7.5K Apr  6  2019 libipx.so.5
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libjail.so.1
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libkiconv.so.4
-r--r--r--  1 root  wheel    34K Apr  6  2019 libkvm.so.6
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libkvm.so.7
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libm.so.5
-r--r--r--  1 root  wheel     0B Aug 28 15:54 libmd.so.6
-r--r--r--  1 root  wheel    98K May 26 14:50 libmlx5.so.1
-r--r--r--  1 root  wheel    21K May 26 14:50 libmt.so.5
-r--r--r--  1 root  wheel   339K May 26 14:50 libncurses.so.8
-r--r--r--  1 root  wheel   375K May 26 14:50 libncursesw.so.8
-r--r--r--  1 root  wheel    89K May 26 14:50 libnv.so.0
-r--r--r--  1 root  wheel    89K May 26 14:50 libnvpair.so.2
-r--r--r--  1 root  wheel   400K May 26 14:50 libpcap.so.8
-r--r--r--  1 root  wheel    16K May 26 14:50 libpjdlog.so.0
-r--r--r--  1 root  wheel   267K Apr  6  2019 libreadline.so.8
-r--r--r--  1 root  wheel    12K May 26 14:50 libsbuf.so.6
-r--r--r--  1 root  wheel   7.5K May 26 14:50 libssp.so.0
-r--r--r--  1 root  wheel   120K May 26 14:50 libthr.so.3
-r--r--r--  1 root  wheel    18K May 26 14:50 libufs.so.6
-r--r--r--  1 root  wheel   9.3K May 26 14:50 libulog.so.0
-r--r--r--  1 root  wheel   5.9K May 26 14:50 libumem.so.2
-r--r--r--  1 root  wheel    77K May 26 14:50 libutil.so.9
-r--r--r--  1 root  wheel    40K May 26 14:50 libuutil.so.2
-r--r--r--  1 root  wheel   117K May 26 14:50 libxo.so.0
-r--r--r--  1 root  wheel   101K May 26 14:50 libz.so.6
-r--r--r--  1 root  wheel   303K May 26 14:50 libzfs.so.2
-r--r--r--  1 root  wheel    33K May 26 14:50 libzfs_core.so.2
-r--r--r--  1 root  wheel   2.0M May 26 14:50 libzpool.so.2


But I've got the 12.0 kernel:
Code: 
dave@mail[~] % uname -a
FreeBSD mail.foobar.com 12.0-RELEASE-p13 FreeBSD 12.0-RELEASE-p13 GENERIC  amd64
dave@mail[~] % freebsd-version -ku
ld-elf.so.1: /lib/libedit.so.7: invalid file format

I'm pretty messed up now. I've got two windows into the system, neither of
them has root. It looks like nothing that isn't staticly compiled will run.
I can use everything in /rescue, but most of it requires root. sudo won't
run. I can't ssh in, so I can't copy files, and I can't reboot to the old
kernel. Even if I could, I'm guessing that not having /lib/libelf.so.* and
other libs would keep the system from booting.

Does anyone know how I might fix this remotely?

If I can't do it remotely, any hints on how to recover to 11.4 or finish the
12.0 upgrade without having to reinstall everything?

Does freebsd-update keep a log file anywhere? I'd like to see what happened.

I would have posted more output, the above is the useful bits I could get from
my scrollback.

Thanks!

- dave
 
The freebsd-update script has a nasty habit of proceeding with installation despite earlier fatal errors. It's happended to me twice that upgrade -r did not download complete tarballs and the subsequent install wrecked the system.

You don't need to downgrade back to 11.4. Copying the necessary files from another 12.0 should be sufficient. Downloading base.tzx and kernel.tzx and carefully copying (without /etc) should also do the job although I haven't tested that path myself.
 
Dave_Berg said:
I've got a server about 50 miles away that failed to upgrade. I wanted to
end up at 12.3-RELEASE, but I figured going to 12.0 first would be a good
idea. I ran the usual:
Click to expand...
12.3-RELEASE doesn't exist yet. It's 12.2-RELEASE.
For the rest, it was a very bad idea because 12.0-RELEASE is anterior to 11.4. See here:
www.freebsd.org

Release Information

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

I admit it's counterintuitive and you're not the only one being so trapped. The worst is that freebsd-update doesn't make any verification and it should.

Had you upgrade to 12.2 and all will working now.

I don't know how to recover from this. You can use the commands in /rescue, they should work as they are statically linked.
 
It's a bit late to help you - but maybe not - this is why it's worth having a test system or VM to experiment with. (And a remote console BMC/iDRAC solution - expensive but worth it at times like these!)

So if you want to see what happened and try and recover from it - set up a VM with 11.4 and repeat the process that you followed - the upgrade to 12.0.

See if you can work out where it goes wrong and how to fix it - on the local machine/VM - and then repeat those steps on the remote machine once you've got a plan that seems to work.

The advantage of the local test machine is that it doesn't matter if you screw it up and have to redo it a few times - it's right there in front of you. You don't have to go for a complete mirror of the remote system for this - just a base 11.4 install, straight to upgrade to 12.0, and see if you can get into the same state and see what gets you out of it. Document everything you do, then redo from scratch to check it really DOES work before you try on the remote machine.

Good luck.
 
Dave_Berg said:
Does anyone know how I might fix this remotely?
Click to expand...
If you can't SSH in: Is that a bare metal install or a hosted VM? With a bare metal install, you have no choice but to come onsite and just redo the install (or have someone else do it for you). With a VM, there might be an option offered to have access to the VM management console.

you mention some kind "windows into the system"
Dave_Berg said:
I've got two windows into the system, neither of
them has root.
Click to expand...

What exactly are you using to connect? Depending on what you use, it just might be possible to gain root access (and you will want to patch that up when you do recover your system!)
 
Matlib said:
Copying the necessary files from another 12.0 should be sufficient.
Click to expand...
The challenge is to do that when su/sudo doesn't work - normal level user only, and the server is remote. Might be able to get someone to plug in a USB stick, but what would the steps be from that point on? Is it possible remotely?
 
Remotely no. I doubt that system would even boot at this point. Booting from a rescue USB stick and connecting to that could be an option but considering the 50 mile distance it's probably not worth bothering.
 
In which case, your options are to either take a 50 mile drive to the location, or ask the shop hosting your server to delete it so that you can start over. And when you do start over, it might be worthwhile to think about a way to more easily recover from upgrade disasters like that. Takes a bit of time and mental effort up front, but I think you'll be satisfied with the ROI on such an investment.
 
Thanks for the help everyone! You confirmed what I thought.

If anyone cares:

  • 12.3-RELEASE was a typo
  • it's bare metal, no VM
  • The 2 ssh logins I had dropped yesterday due to a power outage at my house
  • Having a test system would have been a good idea
  • I forgot that 11.4 came after 12.0, I just wasn't thinking. Shame on me for not reading the release notes for both versions.
  • it would be great if freebsd-update tried a bit harder to keep this from happening. I'll look into it and submit a fix if possible.
  • I've got backups, I was just trying to avoid a trip to the data center. But this point in the pandemic, a day there will be like than a vacation.
And my biggest lesson is to always use:
freebsd-update <options> | tee /path/to/upgrade.log

and don't use sudo for upgrades on remote hardware.

-- dave
 
There used to be an effort to move core FreeBSD to pkg(7), which is far more robust.

If ever freebsd-update fetch or upgrade gets interrupted or dumps any errors to the console you must remove all temporary files:

Code: 
find /var/db/freebsd-update/files -type f -exec rm {} +

and restart. In fact on some systems I just launch this by default before issuing any of those two commands. Otherwise you may end up with corrupted tarballs and eventually nuke the operating system.
 
Matlib said:
There used to be an effort to move core FreeBSD to pkg(7), which is far more robust.
Click to expand...
Sorry, but I believe I have to challenge this... FreeBSD was using pkg-ng since FreeBSD 10-RELEASE (Gleaned from pkg(7)), and pkg was in the base system since FreeBSD 9.1-RELEASE. Per ports(7) the ports collection has been around since FreeBSD 1.0. There was never an effort to move FreeBSD to pkg(7)... There was an effort to improve the pkg (pre-compiled ports) management, true, but not to move. Ports and packages are here to stay, and I like that about FreeBSD.
 
I was intrigued enough to have a look. So I've got in the same state, but got one root ssh open, and one "normal" level user.

As root, the fetch command works - doesn't seem to like https but I can do a fetch from a basic http site. So in theory I might be able to fetch the 12.0 binary files somewhere locally (i.e. set up an http server with the 12.0 binary files, then use fetch to pull them down to the broken machine), and update the system in place with those files? :-/

tar gives a bus error, so the remote files would have to be vanilla, I think - no tar or compression (but gunzip and bzip2 are in /rescue so maybe OK).
 
Took a look at the wiki myself, and the way I'm reading it is that 'Compiling kernel updates with your own options is reliable, but cumbersome. Can those kernel updates be provided as easy-to-install pre-compiled packages?'. There's nothing in that wiki to suggest that compiling your own components is going away. It's here to stay as an open, viable option that is actively maintained.

FreeBSD updates work very differently from Windows Update. Not realizing that (and blindly diving into freebsd-update fetch without doing some very thorough homework on your end) will reliably invite upgrade disasters. It's not impossible to use existing system utilities to line things up properly, so that updates happen without breaking anything in the process, but that is a Napoleonic venture that I'm not quite willing to touch. I just sit back, drink tea, and notice a crowd of forum postings 'I used freebsd-update, now my system is completely messed up, please help!'.

FWIW, I've gone through a few upgrade disasters myself. On my end of things, it was dependency hell that I brought on myself trying to upgrade KDE. It's been many years, and never once was I able to upgrade KDE without having to do a clean install of the entire machine every single time. I think I may be getting close this time with Poudriere and ZFS, but even that takes a lot of studying and lining up options.
 
You must log in or register to reply here.
Back
Top