everything is "invalid file format" after upgrade from 11.4 to 12.0-RELEASE

Sounds like you are making good progress. And have good backups!

It's worth (on a "quiet" day) having a play with the boot media and learning how to mount the partitions etc. in case you ever have this sort of situation again - it's difficult to keep calm when your head is on fire!

Hope your next update attempt goes smoothly - I've always found freebsd-update to be trouble-free (as with most things on FreeBSD). But whisper I do always try a few things on test boxes first - never quite 100% the same as production but a good indicator.
 
I have a really dumb backup system, and by that I mean it doesn't try to do anything cute. I still dump for just these stupid reasons. It's a blunt instrument and it's easy to understand, and when it breaks, it tends to break in a way that makes sense.

And despite first impressions from this fustercluck, I do test things frequently. I've done this exact upgrade on multiple machines before without a problem, but none of them are the flagship server, so it's never quite apples-to-apples. You do what you can with the resources you have.

I'm a little bummed that I still don't know what went wrong in the first place, but I am delighted that my server seems happy and mostly normal. I'm sure I'm missing a couple of config things that will become clear over time, but it's basically the same guy I knew yesterday.

I didn't even lose user or /usr/local data. Very, very lucky!

Also this is a good advertisement for the old school method of partitioning drives that has fallen out of fashion. Having / and /usr on their own partitions was a lifesaver!

Oh, and it turns out I wasn't totally stupid in another way. The FreeBSD installer allows you to label the partitions, and I gave them sensible names a million years ago so I didn't accidentally blast /var instead of /usr or something like that. The installer was nice enough to show me those labels, so it was total warm fuzzies when I saw logical things in the right places.
 
Sounds like a good save! Hopefully upgrade goes smoothly when you try again but as you say then it will be a bit of a head-scratcher as to what happened? Solar flares? Aliens?
 
I'm not bothering with 12.x as 13.0 is already at p3, so it'll save a lot of work in 2024 to go straight to 13 now.

Don't forget to read the Release Notes for both 12 and 13 first though, to avoid any unpleasant surprises.
 
The only reason I went to 12 instead of 13 is that I'd moved so many of the test machines to 12 first, and they've been stable for a while. I suppose trying to be conservative doesn't always work out, although whatever went wrong probably had nothing to do specifically with 12. Probably.

I'll report in when I do the 12.2 update. Might be a little while. I had a little window when I had some time to do this and thought it wouldn't take more than an afternoon. At least I've freshened up my disaster recovery skills.

The system seems to be doing fine, so I don't think it was a hardware issue related to the drives, but you never know.

Thanks again for everyone's help. Blood pressure normal now.
 
Hi there,


I've just came to a similar issue but with root on ZFS system:

root@logpmbackup01p:/usr/home/rod # freebsd-version
11.4-RELEASE-p12


Then:
root@logpmbackup01p:/usr/home/rod # freebsd-update upgrade -r 12.0-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 11.4-RELEASE from update1.freebsd.org... done.
............ la la la ......
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".


To followed by:
root@logpmbackup01p:/usr/home/rod # freebsd-update install
src component not installed, skipped
Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.
root@logpmbackup01p:/usr/home/rod # reboot


An then, the horror:
root@logpmbackup01p:/usr/home/rod # freebsd-update install
src component not installed, skipped
Installing updates...Bus error (core dumped)
Bus error (core dumped)
............. again and again ........
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
.............. many times ...........
chflags: ///lib/librss.so.1: No such file or directory
chflags: ///lib/libzfs.so.3: No such file or directory
chflags: ///usr/lib/libarchive.so.7: No such file or directory
.............. a bunch of those .....................
chflags: ///var/db/etcupdate/current/etc/syslog.d/ppp.conf: No such file or directory
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libedit.so.7: invalid file format
.............
done.



It doesn't reboot any more on kernel or kernel.old

There are more people with the same issue:


I don't have hands on the server, but I'll try to get someone to boot it from a 11.4 USB to see whats there.

Any suggestions anyone?
 
I confirm the problem from 11.4-RELEASE to 12.0-RELEASE. I tried in a VM and then, the system booted in single user mode with an error concerning ld-elf.so.1. For me, this occurs after the second freebsd-update install at reboot but no Bus error.

I can select /rescue/sh as shell. But there, I have no access to a functional freebsd-update.
Hope you have a boot environment from which you can start your system.

That being said, there is no problem to upgrade from 11.4-RELEASE to 12.2-RELEASE.

I think that the point is that 12.0-RELEASE is anterior to 11.4-RELEASE. December 11, 2018 vs June 23, 2020. See here.

In fact, you tried to upgrade to a previous version of FreeBSD.
 
I've faced the same issue recently with the same set of actions:
- freebsd-update -r 12.0 upgrade
- freebsd-update install
- shutdown -r now
- freebsd-update install # here I've got the same log of core dumps...

I would like to post the way I fixed it in case it might be useful for someone. My target was to get back to 11.4 I had before in order to do upgrade right to 12.2 after this 11.4>12.0 failed manoeuvre. And this is not about remote fix, definitely.

- I booted the host from 11.4 USB memstick
- mount /dev/<root-fs-in-danger> /mnt # you will have to mount other FSs if it's split onto /var, /usr etc in order to get complete "system tree"
- cd /mnt/var/db
- cp -rp freebsd-update freebsd-update.bak # let's do some backup of freebsd-update state
- cd freebsd-update
- ln -s install.<?> "$(echo /mnt/ | sha256 -q)-rollback" # find out the name of your install.<?> dir
- freebsd-update -b /mnt/ -d /mnt/var/db/freebsd-update/ -f /mnt/etc/freebsd-update.conf rollback
 
RodrigoC and floodway please, was there the false assumption that a major upgrade must always be to a .0 version?

(<https://docs.freebsd.org/doc/12.2-R...ding-freebsdupdate.html#freebsdupdate-upgrade> for 12.2-RELEASE lacked clarity.)
I personally do not remember any official statement regarding this. I guess, it's not a concern if an instance tries to follow new releases constantly, and if it's outdated for longer period than it needs more caution (yeah, undetermined "caution").

The best answer, I think, could be an official statement from a FreeBSD dev, who better understands this situation from dev cycle perspective and upgrade tech process details. And added to the official documentation you referred. For example, it could be a recommendation like "always do major upgrades to <X+1>.<Latest>", if it really is a correct statement.

If it's hard to reach a dev, I guess, it could be sent as a bug request to patch the doc respectively - it may trigger required discussion, and real change eventually.
 
freebsd-update(8) can upgrade from any[*] -RELEASE version to any other -RELEASE version. You do NOT need install any of the intermediate versions.

[*] Not entirely correct, there's is a 'lowest' version, before 6.2 freebsd-update(8) didn't exist. But I wouldn't recommend upgrading from 6.2-RELEASE to 13.0-RELEASE any way, it's just too big of a step up. It's perfectly fine however to upgrade from 10.3 to 13.0 or 11.2 to 13.0 skipping over all the 12.x versions.

There has been one exception, at some point in time freebsd-update(8) had a bug that prevented it from upgrading to 11.x (or above). To fix that bug you need to have at least the latest patch version of a -RELEASE.
 
Ahhh... the horror stories about freebsd-update(8) that I constantly see on these forums are enough to scare me away. I personally concluded that this kind of thing, it's OK to practice them in a throwaway VM if you have the time and the hardware to do that. I'd want to study my ass off about freebsd-update(8), and make sure the versions match, and all other pitfalls are avoided - BEFORE diving off a cliff like that. :eek:
 
Cross-reference: Can I upgrade an old computer that isn't used that much from 12.1 to 12.3 directly or is skipping point releases not allowed? : freebsd

… horror stories about freebsd-update(8)

The situation is less horrible than might be imagined.

Some stories relate to problems before or after use of the tool:
  • not a problem with the tool.
Some almost certainly relate to misuse of the tool:
  • not using the tool as instructed in the FreeBSD Handbook.
… People that don't have problems don't post about it.

True, and (generally) people that do have problems very often do not disclose what's required for readers to appreciate the true nature of the problem.

Particularly with freebsd-update, I always try to read between the lines (including prior posts/topics by the reporter) to guess whether there was misuse.

I recently went through some open bugs in Bugzilla, resulting in a closure or two.

Most memorable (already closed) were the bugs that led to erratum FreeBSD-EN-21:08.freebsd-update. Other mentions of freebsd-update can be found at pages such as <https://bokut.in/freebsd-patch-level-table/> …
 
Sorry for the Necropost, but Wisdom of the Ancients. This is for anyone that runs across this situation now and in the future with any builds.

I had the same situation as the OP. I bounced from 11.4 to 12.0 on my way to 12.3, I should have known better, but I digress. I managed to fix it without reinstalling on a remote 1000+ connection server, 1000 miles away, in a Linux shop datacenter. (So even if I paid $200/hr to fix it, they wouldn't be able to via USB.)

When you hosed the system and caused the core dumps and ld-elf.so.1: /lib/libmd.so.6: invalid file format errors, you should have at least 1 shell logged in as root. And your services should be running. You can't launch any programs outside of /rescue/. Whatever you do, don't disconnect or reboot the system.

So what happened? 11.4 is newer, so the /lib/ files are newer than what 12.0 tried to install. So the 12 /libs/ didn't install. I suspect this will happen anytime you jump from a newer old build to an older new build. You need to install 12.0 /lib/ files, which means you need to grab 12's base.txz.

Sadly, I don't see anything in /rescue/ that can do that. (It would be wonderful if they included wget or some sort of recovery program in there.)

This is where your services come into play. Luckily for me, I still had access to PHP, with a 15Mb upload limit. Yay! I download Base.txz on my local system and extracted it. The first job is to get sshd working in case of network failure.

Run sshd -t. It should spit out some broken libs. Find them in your local extraction of base and upload them to your PHP uploader (you either wrote one using /rescue/vi or you got something like forum software installed. Alternatively, if you have mail service still going, an email could work.). Move those libs to /lib/. I think I needed libssl.so.111, and I also brought in /lib/libedit.so since everything was complaining about it.

Run sshd, connect to it and log in. If you can, great! You're ready for the headlong charge.

You can either try to get wget running or upload /lib/ piecemeal. I chose the latter in 2 tar files. Make sure you have those files all in one folder, say 12BaseLib, as you're going to get only one chance at this. Using /rescue/cp -R copy everything in your uploaded 12BaseLib folder to the actual /lib/ folder. You're going to be kicked off shell as EVERYTHING will crash on the system because you're copying over libc and what have you. /rescue/cp should still do its job, though. At this point, you wait 10minutes. Then you send a tech support ticket to reboot the system. Wait another 10 minutes, and hopefully, you can log in.

If you can, you're almost home.

Now you need to Freebsd-update to 13, as that will copy over the most files. You'll run through that entire system, a few more things will fail, like libregex, but the install should go through. Sshd and a few other programs will break again. You will have to manually merge files because diff is broken. When you finish that, it will install, but there will be a few errors. At this point, a few programs will work, notably wget. Download the 13.0 base.txz, extract it to 13Base or something. You'll need to copy over the files you need for sshd to make sure it works again. Then using /rescue/cp move over the stuff in 13Base/usr/lib to /usr/lib and you'll also need to do the /share/ folder. This action shouldn't crash, but it might. At this point, you should be able to run pkg-static upgrade -f, and nothing should break. Finish up the install, verify that sshd works again, reboot. Clean up any config issues you have now with the update. (PHP 8, ugh)

That's what worked for me. All my services are running, the system reboots, no errors in my log files.

Keep 13Base.txz file around if you run into any missing or malformed files.

In the future, always bounce to the latest minor builds of Release. And download base.txz files to the system if you have the space. They're pretty handy if you screw something up.
 
Back
Top