Upgrade disaster 11.2-RELEASE -> 12.0-RELEASE

ahhyes

Member

Reaction score: 7
Messages: 65

Hi All,

Well, this is a first. I have been blessed with years of FreeBSD upgrades going flawlessly until I went to upgrade a machine last night using the freebsd-update utility.

All seemed to go well, first phase (which from memory installs the kernel portion of the upgrade) I did the freebsd-update install command which completed without any errors, then told me to reboot and run that same command again, and that's where it all went bad. I rebooted and the machine no longer boots and bootloader seems to crash (I have attached a photo)... Not sure what to do from here.. Any suggestions?

btxcrash.jpg
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

I should mention I do have a ZFS pool on the machine, but the OS is not installed on it. FreeBSD is installed on a regular hard disk formatted with UFS and boots from there.
 

johnblue

Well-Known Member

Reaction score: 40
Messages: 309

FreeBSD is installed on a regular hard disk formatted with UFS and boots from there.
If you are like me I reach a point of diminising returns when an OS tanks. Plus, my OCD has a hard time trusting it. I would get another hard drive and install fresh then mount the old hard drive and cp off what you need.

After prod is back up do your postmortem at your leisure.

Good hunting.
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

Any ideas what may have gone wrong? I see the boot loader print a list of drive letters briefly (which seems normal), then boom. crash. It never loads the kernel from what I can see. I find it strange the screen is also corrupted, theres flashing ascii characters too in those verticle colored bars (though the camera didnt catch it).
 

k.jacker

Aspiring Daemon

Reaction score: 344
Messages: 590

I can't make anything of the output on your screen, but the looks of your screen point towards a severe hardware problem.

It could be random that it hit you now, but if the computer had long uptime, since last reboot, maybe something build up that didn't show.

I would in the first step, take a pendrive with the FreeBSD installer, to see if the computer will boot. If yes, I'd probable look at gpart show to see if it still shows a partition table and maybe rewrite the EFI partition (you booted in EFI mode i guess because it shows in the photo).
If it won't boot from the installer either, see if you can enter the BIOS and load defaults and then check the battery.
If you can't get into the BIOS, check RAM, PSU and so on...
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

I would in the first step, take a pendrive with the FreeBSD installer, to see if the computer will boot. If yes, I'd probable look at gpart show to see if it still shows a partition table and maybe rewrite the EFI partition (you booted in EFI mode i guess because it shows in the photo)
I've never enabled EFI mode to my knowledge, how can you tell its EFI? :)

Just strange. This machine spends most of its time powered off, it is primarily a NAS which hasnt exhibited any signs of hardware failure to date. I guess its possible I just suddenly got hit with a hardware failure, but I would have to have been pretty unlucky!
 

ShelLuser

Son of Beastie

Reaction score: 1,696
Messages: 3,512

The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?

What happens if you drop down to the ok> prompt (press escape) and then issue the lsdev command? Does that list your drives (at least the ZFS pool)?

Also: what happens when you use the menu to boot the previous kernel (kernel.old)?
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?

What happens if you drop down to the ok> prompt (press escape) and then issue the lsdev command? Does that list your drives (at least the ZFS pool)?

Also: what happens when you use the menu to boot the previous kernel (kernel.old)?
It feels like bootloader to me (but I could be wrong of course), I only say this because of the "BTX halted" message, and secondly it happens so fast. This system boots from good old spinning rust (which is slow compared to the likes of flash storage), so theres always those few seconds where it loads the kernel (that seems noticably absent in this case). I quickly see the loader print a list of drive letters and its an instant crash.

I will try those commands and see what I find (at work at the moment so cant check right now)
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

the CS:EIP pair in that register dump is rather alarming - assuming this is for protected mode, it will be codeselector and offset - that looks invalid to me and may be the reason for the crash.
 

Wamphyre

New Member

Reaction score: 4
Messages: 11

I suffered the same problem after update. Just disconnect any external drive (USB drives, USB HDDs, etc) and boot again
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

I suffered the same problem after update. Just disconnect any external drive (USB drives, USB HDDs, etc) and boot again
No external drives connected to this system. I wonder if the drive ordering has gotten changed somehow.. I wont know until I try the commands that 'shelluser' suggested in their reply. So you had the same type of crash as me?
 

Wamphyre

New Member

Reaction score: 4
Messages: 11

Yes, after update from 11.2 to 12-RELEASE I experienced the same "BTX Halted" error. Tried to boot again without any external drive connected, and it worked (with 11.2 wasn't any problem with that)
 

drhowarddrfine

Son of Beastie

Reaction score: 1,123
Messages: 3,059

The only thing we know you did was the last of five or more steps involved with updating. What did you do previous to the install part?
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

The only thing we know you did was the last of five or more steps involved with updating. What did you do previous to the install part?
I didn't get far into the upgrade before this hit (from memory this was the sequence of events):

freebsd-update -r RELEASE-12.0 upgrade

<it downloaded a whole lot of stuff as usual, then I was eventually prompted about merging some config in /etc, nothing eventful there>

freebsd-update install

<this part completed with no error and advised me that to continue the upgrade, to reboot and then run freebsd-update install again (presumably to install the userland stuff)>
<I rebooted and hit the error as shown in the OP>
 

edenist

New Member

Reaction score: 5
Messages: 9

What CPU is in this system if you don't mind me asking?

Over the past few months, I've had all sorts of kernel problems stemming from meltdown/spectre mitigations [the OS implementations along with all sorts of microcode combinations].

I believe there are also some NUMA changes in 12.0, and the errata also mentions boot problems with J1900 CPUs. I don't suppose any of these apply to your case?
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

What CPU is in this system if you don't mind me asking?

Over the past few months, I've had all sorts of kernel problems stemming from meltdown/spectre mitigations [the OS implementations along with all sorts of microcode combinations].

I believe there are also some NUMA changes in 12.0, and the errata also mentions boot problems with J1900 CPUs. I don't suppose any of these apply to your case?
It is a Xeon E5-1650V2 from what I can recall.
 

tedbell

Active Member

Reaction score: 43
Messages: 109

The upgrade destroyed my machine twice as well as well as a host of other problems.

The first time was an upgrade and with my USB drives connected I got BTX halted. The issue is NOT on my end because I can boot fine with 11.2 with UFS on my boot drive.

The second time was a clean install with the USB image. Installation would not detect my USB network adapter like 11.2 would. Another BTX failure with drives connected. Then it took hours to connect to the internet because
Code:
ifconfig wlan0 create
would not work as it did in 11.2. Then when I somehow managed to connect all the mirrors were messed up and I couldn't even download pkg.

The third time was another upgrade from 11.2 and I couldn't boot period with or without drives connected.

Total disaster. Sticking with 11.2
 

Datapanic

Well-Known Member

Reaction score: 172
Messages: 354

I prefer to either run fresh installs or upgrades on the FreeBSD RC's on my stuff which I can easily backup and restore first, just in case it doesn't work before doing it with *-RELEASE next versions on my production systems. Of course, always have snapshots/backups that work before doing it.

Meanwhile, I'm running Poudriere on a temp server to create new packages for my 12.0 preferences before I can install or upgrade any of my other systems to 12.0.

Good luck with your upgrades!
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

A few changes to the bootloader there:
https://www.freebsd.org/releases/12.0R/relnotes.html#boot

Not much that can go wrong with all those changes is there :-/.. I've just written a 12.0-RELEASE memstick image to a usb memory stick and will test this when I am home. I'm curious to see if that even boots. If it doesnt, then it looks like 12.0 has introduced some incompatibility with my hardware. I'm willing to wager a small amount of money that a 11.2-RELEASE image will boot if 12.0-RELEASE fails (gut feeling).
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

Looking at some of the other forum threads.. It seems I am not alone with 12.0 failing to boot.
 

VladiBG

Aspiring Daemon

Reaction score: 222
Messages: 578

Try to boot from 12.0-RELEASE LiveCD. If it's boot normally then try to update your bootstrap code.
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?

What happens if you drop down to the ok> prompt (press escape) and then issue the lsdev command? Does that list your drives (at least the ZFS pool)?

Also: what happens when you use the menu to boot the previous kernel (kernel.old)?
lsdev doesnt appear to do anything: "No lsdev on 0:ad(0p2)"

Trying to do a boot kernel.old from the boot loader is telling me "gptboot: No kernel.old on 0:ad(0p2)"

I tried booting from a 12.0-RELEASE memstick image, on a USB drive, it also crashes with the same error...
 

ShelLuser

Son of Beastie

Reaction score: 1,696
Messages: 3,512

lsdev doesnt appear to do anything: "No lsdev on 0:ad(0p2)"

Trying to do a boot kernel.old from the boot loader is telling me "gptboot: No kernel.old on 0:ad(0p2)"
Ok, now that's bizarre. The first error makes me believe that the bootloader might have been damaged or incorrectly installed, yet the second error would also point at your root filesystem.

I tried booting from a 12.0-RELEASE memstick image, on a USB drive, it also crashes with the same error...
At this stage I'd try to boot with a 11.2 image just to check what that says. Because something is seriously bad here.
 
OP
OP
A

ahhyes

Member

Reaction score: 7
Messages: 65

At this stage I'd try to boot with a 11.2 image just to check what that says. Because something is seriously bad here.
I am putting my money on that working. lol. Downloading it now. I'll report back on the results.
 

VladiBG

Aspiring Daemon

Reaction score: 222
Messages: 578

If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also try to update your BIOS.
 
Top