How Do I Recover My Zero-Length Shared Libraries After freebsd-update?

Esteemed Colleagues,

After running freebsd-update to upgrade my system from 11 to 12, I can no longer boot my system, because there are zero-length shared library files in /lib.

If you are wondering why I waited till March 2025 to upgrade my system from 11, it is because This Always Happens. Whenever I upgrade my system, it always -- always -- costs me at least a week of my life. But after updating my ports tree with portsnap (because that's what we use in version 11), I could no longer build any ports, because some genius had decided to put lines in their makefiles that begin with a hyphen, on which the native make command choked. So I had to update my system to a version with a make command that doesn't die with an "operator expected" message when it sees one of those lines in a makefile beginning with a hyphen.

After the 2nd "freebsd-update install" and subsequent reboot, the boot fails. The failure results in the message: "Enter pathname or return for /bin/sh" (words to that effect, I don't have the message in front of me). I type return for /bin/sh, and I cannot execute it; the error message is "ld-efl.so.1: /lib/libedit.so.7: invalid file format". /bin/csh also fails, complaining about /lib/libcrypt.so.5. I eventually got in with /usr/local/bin/bash, whereupon a fortunately-still-working ls command informed me that /lib/libedit.so.7 is an empty file. I looked for other empty files in /lib (although not, of course, with "ls -s /lib | sort -nr", because that would be too easy; the "sort" command also fails, due to the zero-length /lib/libm.so.6) and discovered that there are 20 zero-length files in /lib, ranging alphabetically from libcrypt.so.5 to libmd.so.6. And surely there must be others in other directories. The freebsd-update command could not have trashed only twenty files, that would be totally uncharacteristic of it.

You might think that the freebsd-update command ran out of disk space, hence the zero-length files. But first of all, if that were true it would be incredibly moronic, for a program as crucial as freebsd-update, on which so much depends, not to notice that its writes are failing. Second of all, it cannot be true. The df command reports that the root filesystem is only at 90% capacity, with more than 7.8 gigabytes available (this is after the update), and don't forget that root has more than 7.8 available gigabytes, because root can run the filesystem up to more than 100% capacity and negative availability. The explanation can only be that the authors of FreeBSD want me to waste my life, and to die with more than half of my dreams unfulfilled. Friends and colleagues -- how do I recover those zero-length files, so I can boot my system and then observe all the other damage that freebsd-update has inflicted on me? (And why isn't /bin/sh statically linked, as it is on all right-thinking systems?) As always, thank you in advance for any and all replies.
 
(And why isn't /bin/sh statically linked, as it is on all right-thinking systems?)
Because we have a /rescue directory filled with "static" executables (it's actually all just one executable, almost everything is hard linked to a single file), in case there's a problem.

And /bin/sh isn't statically linked on Debian, and I'm pretty sure it isn't on RedHat either (can't check right now), or on plenty of other Linux systems, or MacOS.
 
I suddenly remembered seeing this before: https://forums.freebsd.org/threads/freebsd-update-broken.74179/

Never figured out in that thread what the cause was though. I'm certainly not ruling out bugs in freebsd-update(8), there have been quite a few bug fixes (Errata Notices) since the release of FreeBSD 11. Speaking of FreeBSD 11, what minor version and patch level does it have?

I'm also not ruling out issues with downloading the patch files, are you passing this traffic through a proxy server perhaps? I've also seen people report issues with so-called "threat protection/prevention" features on some firewalls, those can mangle the downloads too.
 
Back
Top