Upgrade disaster 11.2-RELEASE -> 12.0-RELEASE

OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also can try to update your bios it may help also.

Its a good image :) I checked the sha512sum before I wrote the image to the memory stick
 

teo

Aspiring Daemon

Reaction score: 45
Messages: 704

Hi All,

Well, this is a first. I have been blessed with years of FreeBSD upgrades going flawlessly until I went to upgrade a machine last night using the freebsd-update utility.

All seemed to go well, first phase (which from memory installs the kernel portion of the upgrade) I did the freebsd-update install command which completed without any errors, then told me to reboot and run that same command again, and that's where it all went bad. I rebooted and the machine no longer boots and bootloader seems to crash (I have attached a photo)... Not sure what to do from here.. Any suggestions?

View attachment 5668

It looks like the system detected Xorg as corrupt and he's not getting past it.
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

11.2-RELEASE boots perfectly....

Ok. So what on earth is 12.0-RELEASE unhappy about on that hardware? Bizarre.

freebsd-update does have a rollback option, will that get me out of trouble here?
 

_martin

Daemon

Reaction score: 496
Messages: 1,336

Hm, Xeon E5-1650V2 is 64b. Are you using 32b version of FreeBSD ? Segment registers suggest you are already in protected mode (CR0 reg would confirm that). CS:IP seems ok, the contents of the mem where it points to is ivalid (0xff -> bad instruction , int 6 ).

Consider using beadm(1) and create new boot environment when doing upgrade. With that you can easily revert/reactivate old env.

UPDATE: ok, I went through your thread again and noticed you mentioned ZFS is not root pool .. never mind my advice on beadm then :)
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

Hm, Xeon E5-1650V2 is 64b. Are you using 32b version of FreeBSD ? Segment registers suggest you are already in protected mode (CR0 reg would confirm that). CS:IP seems ok, the contents of the mem where it points to is ivalid (0xff -> bad instruction , int 6 ).

Consider using beadm(1) and create new boot environment when doing upgrade. With that you can easily revert/reactivate old env.

As far as I know the system has always been running FreeBSD amd64
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

hm.. is there any way to remount / from single user mode when booting from a USB drive? it would be nice to remount / and do a rollback?
 

Wamphyre

New Member

Reaction score: 4
Messages: 12

I tried (just for testing) a new 12.0-RELEASE installation with external disks attached. Boot fails with "BTX Halted" error.

Removing external drives = It boots well (looks spooky)

I'm still thinking that your problem is related with your disks or BIOS configuration (Try from IDE to AHCI or viceversa)
 

Wamphyre

New Member

Reaction score: 4
Messages: 12

Definitely, there's a bug on the BTX loader in FreeBSD 12.0-RELEASE (I suppose). With 11.2 there's no problem with boot process with or without external drives connected
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

Any ideas how I can revert back to 11.2 at this point? I can boot with the 11.2 usb image but / is already mounted and I can only remount /dev/ada6p2 to another location.
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

can you boot with the kernel.old?

boot kernel.old doesnt seem to work sadly.. the bootloader is screwed.

if I mount the hard disk after booting from an 11.2-RELEASE memstick image I can see all the files are there under boot folder. It suggests the hard disk contents itself are fine and the bootloader is screwed up.
 

VladiBG

Daemon

Reaction score: 686
Messages: 1,409

boot with the LiveCD create a directory /tmp/root and mount your freebsd-ufs partition there. Setup your network and make a backup of your files. Then you can reinstall 11.2-RELEASE

EDIT:
After you make a backup you can try this (This may be bad advice so try on your own risk)

boot with the 11.2-RELEASE livecd and setup your network
dhclient hn0 # or use ifconfig to set your ip address/gateway/dns
setenv UNAME_r 12.0-RELEASE # this will trick freebsd-update that we are running 12.0-RELEASE
mkdir /tmp/root # create a mount point
mount /dev/ada6p2 /tmp/root # mount the root partition from your hard-disk
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ upgrade -r 11.2-RELEASE # upgrade(downgrade) 12.0-RELEASE to 11.2-RELEASE
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install # Install 11.2-RELEASE kernel
Do not reboot run again the install
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install # Install 11.2-RELEASE userland
reboot
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

boot with the LiveCD create a directory /tmp/root and mount your freebsd-ufs partition there. Setup your network and make a backup of your files. Then you can reinstall 11.2-RELEASE

EDIT:
After you make a backup you can try this (This may be bad advice so try on your own risk)

boot with the 11.2-RELEASE livecd and setup your network
dhclient hn0 # or use ifconfig to set your ip address/gateway/dns
setenv UNAME_r 12.0-RELEASE # this will trick freebsd-update that we are running 12.0-RELEASE
mkdir /tmp/root # create a mount point
mount /dev/ada6p2 /tmp/root # mount the root partition from your hard-disk
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ upgrade -r 11.2-RELEASE # upgrade(downgrade) 12.0-RELEASE to 11.2-RELEASE
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install # Install 11.2-RELEASE kernel
Do not reboot run again the install
freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install # Install 11.2-RELEASE userland
reboot

Thank you :)

The update process didnt get as far as installing the userland for 12.0-RELEASE - it screwed up after the first phase which was installing the 12.0 kernel (and I am guessing that phase also "upgraded" the bootloader), reverting that phase would probably be enough to fix the system.
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

Is it possible this might work (after mounting the disk under /tmp/root):

freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ rollback
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

Is it possible this might work (after mounting the disk under /tmp/root):

freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ rollback

Well I answer my own question, it wont work, freebsd-update complains it cant find the rollback dir, so I ran the same command under truss and here is the result:
rollback.jpg


So truss reveals what freebsd-update is looking for when it tries to rollback.. And the subsequent find command I did shows what rollback directories actually exist.. Interestingly the rollback dir has the right date on it for when I did the upgrade, but freebsd-update seems to have other ideas on what it wants to look for.. I guess this is the end of the road for me?
 

VladiBG

Daemon

Reaction score: 686
Messages: 1,409

Your boot loader shouldn't be changed by freebsd-update. I think that rollback is only for the updates and won't work for major version changes where you also merge some configuration files. But after you cannot boot from the 12.0 livecd then the bootloader from 12.0 is not working for you. You still can manualy update it using
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada#

edit:
if everything else fail then just install a fresh copy of 11.2-RELEASE and restore your files from the backup.
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

Your boot loader shouldn't be changed by freebsd-update. I think that rollback is only for the updates and won't work for major version changes where you also merge some configuration files. But after you cannot boot from the 12.0 livecd then the bootloader from 12.0 is not working for you. You still can manualy update it using
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada#

edit:
if everything else fail then just install a fresh copy of 11.2-RELEASE and restore your files from the backup.

Thank you for the help so far :) Appreciate it.

There absolutely were boot loader changes:
https://www.freebsd.org/releases/12.0R/relnotes.html#boot

Its starting to look like I will have to do what you suggested earlier, get the network running manually and get the data off the drive to somewhere safe and reinstall...
 

_martin

Daemon

Reaction score: 496
Messages: 1,336

truss output at least confirms it's a 64b version. Interesting that dump in the first pic shows 32b regs on 64b (even in early stage of the boot ; but I guess it's more interesting to me ..).

If I were you I'd go for fresh install. It maybe a bit pain to rebuild/reinstall, but at least it won't be a mess of old/new stuff.
Steps are easy actually: you boot from the memstick/cd , mount the root partition to /mnt/, do a backup:

Code:
# cd /mnt
# mkdir old
# mv * old
Warning: This effectively messes up your original / as it moves it to /old. However you don't loose data and you can still access them when needed. If you have some other, non-standard directories there, you could leave them ( basically you need to have fresh /bin /sbin /var /boot /lib /lib32 /usr /etc )

Now just to a fresh install of FreeBSD 11.2:

Code:
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/doc.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/lib32.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/kernel.txz
# sh
# for i in base doc lib32 kernel; do tar --unlink -C /mnt -xvPjf ${i}.txz ; done
# cp /mnt/old/etc/fstab /mnt/etc
# cp /mnt/old/etc/rc.conf /mnt/etc/
# cp /mnt/old/boot/loader.conf /mnt/boot/
And reboot to it ..
 

gkontos

Daemon

Reaction score: 490
Messages: 2,166

That might be a very long shot! I had the same problem with a version of 10.X in one server while trying to boot with a USB drive connected. I read that you do not have any...

Can you disconnect all your disks and try booting off from the FreeBSD 12.0-RELEASE memstick?
 
OP
A

ahhyes

Member

Reaction score: 8
Messages: 75

truss output at least confirms it's a 64b version. Interesting that dump in the first pic shows 32b regs on 64b (even in early stage of the boot ; but I guess it's more interesting to me ..)

I would have expected to see the likes of RAX, RBX etc too... Maybe the register dump for BTX only displays the 32 bit portion of the CPU registers.

And, thank you for your help too, I will look into the reinstall steps given. I am also considering just backing up what I can and reinstalling the OS. The most important stuff is on my zfs array which should be unaffected by a reinstall as long as I backup the critical configuration on the main drive used by the OS first.
 
Top