Solved Server not reachable after upgrade to 13.2-RELEASE

ultrageranium

ultrageranium

TL;DR running out of space while upgrading led to a situation while apparently successfully upgraded, the system became unbootable due to many 0 bytes binaries in the boot folder. The fix was to chroot, find these files and replace them manually.

Hello, I recently upgrade from 13.1-RELEASE-p9 to 13.2-RELEASE. The process, namely freebsd-update -r 13.2-RELEASE upgrade, failed because I did not have enough free space. After making some space, I restarted and did not have any errors. However after rebooting the machine has been unreachable.

The data-centre where the machine is located provides a 13.0 rescue image and I used it to successfully mount the disk and chroot in the system. Looking at/var/log/messages, the last recorded messages are reboot related, so FreeBSD is definitively unable to start. Still inside the chroot, freebsd-version indicates that I'm on 13.2-RELEASE-p2.

Is there a possibility to somehow force reinstall 13.2-RELEASE? I'm wondering if somehow some important files got corrupted? Like the kernel?

I tried the following commands:
- freebsd-update --currently-running 13.1-RELEASE-p9 -r 13.2-RELEASE upgrade
- freebsd-update -F -r 13.2-RELEASE upgrade

However, they both fail the same.

Code: 
src component not installed, skipped                                               
Looking up update.FreeBSD.org mirrors... 2 mirrors found.                           
Fetching metadata signature for 13.1-RELEASE from update1.freebsd.org... done.     
Fetching metadata index... done.                                                   
Fetching 1 metadata patches. done.                                                 
Applying metadata patches... done.                                                 
Fetching 1 metadata files... done.                                                 
Inspecting system... done.                                                         
                                                                                    
The following components of FreeBSD seem to be installed:                           
kernel/generic world/base                                                           
                                                                                    
The following components of FreeBSD do not seem to be installed:                   
kernel/generic-dbg world/base-dbg world/lib32 world/lib32-dbg                       
                                                                                    
Does this look reasonable (y/n)? y                                                 
                                                                                    
Fetching metadata signature for 13.2-RELEASE from update1.freebsd.org... done.     
Fetching metadata index... done.                                                   
Fetching 1 metadata patches. done.                                                 
Applying metadata patches... done.                                                 
Fetching 1 metadata files... done.                                                 
Inspecting system... done.                                                         
Fetching files from 13.1-RELEASE for merging... done.                               
Preparing to download files... done.                                               
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory 
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory
/usr/sbin/freebsd-update: cannot open files/.gz: No such file or directory         
Attempting to automatically merge changes in files... done.                         
                                                                                    
The following file could not be merged automatically: /etc/ssh/sshd_config         
Press Enter to edit this file in vi and resolve the conflicts                       
manually...                                                                         
                                                                                    
                                                                                    
The following file will be removed, as it no longer exists in                       
FreeBSD 13.2-RELEASE: /etc/defaults/devfs.rules                                     
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/defaults/rc.conf                                         
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/devd/hyperv.conf                                         
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/disktab                                                 
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/inetd.conf                                               
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/freebsd.cf                                         
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/freebsd.mc                                         
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/freebsd.submit.cf                                   
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/freebsd.submit.mc                                   
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/helpfile                                           
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/sendmail.cf                                         
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mail/submit.cf                                           
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/motd.template                                           
Does this look reasonable (y/n)? y                                                 

The following file will be removed, as it no longer exists in
FreeBSD 13.2-RELEASE: /etc/mtree/BSD.debug.dist                                     
Does this look reasonable (y/n)? y

At this point I just abort the script because it clearly makes no sense to go on.

Any help would be greatly appreciated!
 
ultrageranium said:
After making some space, I restarted and did not have any errors.
Click to expand...
Crucially, when did you reboot? After the fist freebsd-update install and then things went south? Or did it happen after the second freebsd-update install?

After the first freebsd-update install only the kernel has been upgraded. You can easily revert this by renaming /boot/kernel.old back to /boot/kernel.

If it happened after the second freebsd-update install things get a little messier. If you have a ZFS system then a boot environment got created with freebsd-update install, and you just select the 'old' (pre-install) boot environment.
 
I am pretty sure that it happened after the first. Unfortunately reverting the kernel did not help, upon closer inspection it's also a complete mess. The kernel file in /boot/kernel.old and /boot/kernel are the same (FreeBSD 13.1-RELEASE-p9 GENERIC, 3298f36b450eb67a5826f055dfee9df7 md5sum), and /boot/kernel is much smaller with a lot of 0 bytes files. Looks like these were created when the disk ran out of space and skipped after restarting the upgrade.

I tried to make a new /boot/kernel with 13.2-RELEASE kernel.txz but it's the same result, does not start. I need to go AFK for a while, if you have more suggestion please let me know. Unfortunately no ZFS to save me. I was thinking to go back in the chroot and try to find all 0 bytes files in case more were to be found...
 
Fixed!

I ran find / -size 0 in the chroot to see if there were more damages and indeed there were a few binaries from the /boot folder that were concerned. I replaced them with matching files from 13.2-RELEASE base.txz and that did the trick. Just to be sure once I managed to boot it again and ssh back in, I also ran a freebsd-update IPS and also spotted another binary that did not match. I suppose that was the last one that was in the middle of being written when the system ran out of space.

If this happens again, running out of space while upgrading, what the best way to force restart from scratch? Given my situation, unless mistaken, it seems the script keeps going on and does not make a difference between a successfully copied file and a failed attempt, so I'd need to find a way to reset its progress/state.
 
You must log in or register to reply here.
Back
Top