Unable to upgrade to 13 from 12

J

juanjico

Hello!

I have been trying for several days to upgrade a 12.3-RELEASE installation (running on ESXi) to 13.3-RELEASE to avoid the sshd vulnerability. I've disabled SSH, so all the process was done using local console at ESXi.

First time tried doing upgrade directly to 13.3-RELEASE using freebsd-update upgrade -r 13.3-RELEASE. Rebooted and the kernel was 13.3, so execute freebsd-update install again to update userland. After about 30 minutes, the server rebooted itself and only show a boot prompt with the error "not ufs". Filesystem is UFS so I thought it might have been corrupted during upgrade. Fortunately, I took a snapshot and went back.

Now, tried to upgrade step by step, going 12.3->12.4->13.0->13.1->13.2->13.3 .... all went well until the upgrade to 13.0 from 12.4. The first reboot went well and show kernel 13.0, but freebsd-update install run started to take several time, i don't know why. Because I was using the local terminal, i can't open another console to check with top what's happening. But, at the ESXi performance tab, I see that 100% CPU usage was happening.

After more than one hour waiting, I rebooted the server and got the same "not ufs" boot screen. Recovered the server with the snapshot again.

Then tried to go from 12.4 to 13.1 and all go well, even installing the userland. But, then rebooted the server and ....

1723640080135.png


What could be the problem ? Every time the server run in 13.x version, I'm unable to reboot the server.

Any help will be really apreciated.

EDIT: While running on 13.1 and not rebooting, the server works. But I found that the disk is not UFS ?

1723643869526.png

1723643907690.png


Best regards,
Juanjo.
 
Not sure, but your freebsd-boot partition is undersized. Mine is 512 KiB.

Anyway, when you reach 12.4, try: gpart bootcode -b /boot/pmbr -p /boot/gptboot -i1 da0

If you get an error with this command, post it. If not, upgrade to 13.3-RELEASE.

You may need to take some room on your freebsd-swap partition, if this doesn't work.
 
covacat thanks, already done

But, still don't work. Applied 13.3-RELEASE kernel and rebooted, and before apply userland updates, tried a fsck:

1723655137836.png


Again, the device is not found. Instead of apply 13.3 userland, tried to reboot again ...

1723655240845.png


Now, the error "not ufs" don't show.

So, I can boot in the new 13.x kernel, and nothing more. The filesystem seems to get corrupted ?
 
Your install comes from ages. I ain't sure there is enough room in your freebsd-partition. /boot/gptboot file on 13.3-RELEASE-p5 is 62018 bytes. It should fit, that said.

Maybe, it's time to consider a complete reinstallation.
 
covacat because as Emrion said, this is an installation from ages. Was a physical server that was virtualized to ESXi. But it's working and using i386 is not a problem for the software running.

I tried to upgrade directly to 14.1-RELEASE and had the same behavior:
1723657607323.png


Reboot and ...

1723657646965.png


I can consider a complete reinstallation, but I'm curious to know whats the problem. Why I can boot in the new kernel only one time .... ?
 
i have no idea why it happens. also no problem with i386 if you use under 3gb of ram just its not as well tested. i suspected some flaw in the fs which is not problematic in 12 but screws up in 13. try to boot 13 from install media and fsck -fy the partition
 
juanjico said:
Because I was using the local terminal, i can't open another console to check with top what's happening.
Click to expand...
You can switch with <alt>+<FN> keys where fn is F1,F2.. (not the function fn key) when using VM console.

tingo said:
what does
Code: 
lsdev
from prompt tell you?
Click to expand...
It's too soon in boot process (no /boot/loader yet), there's no lsdev at this stage.

Partition size is big enough to fit it. I actually tested i386 12.3->13.3 upgrade and it went fine without even touching bootcode.

juanjico said:
The first reboot went well and show kernel 13.0, but freebsd-update install run started to take several time, i don't know why.
Click to expand...
Try to switch to different virtual console and check what is happening here.

That "not ufs" is weird. Any chance you are executing other commands you didn't mention?
 
covacat said:
i suspected some flaw in the fs which is not problematic in 12 but screws up in 13. try to boot 13 from install media and fsck -fy the partition
Click to expand...
For fsck(8) better use the version of 13.1-RELEASE or later, that uses the latest version by Kirk McKusick (PR 255979).

However:
juanjico said:
covacat because as Emrion said, this is an installation from ages. Was a physical server that was virtualized to ESXi. But it's working and using i386 is not a problem for the software running.

I tried to upgrade directly to 14.1-RELEASE and had the same behavior:
Code: 
  [...]
root@soch:~ # fsck /
Can't open /dev/da0p2: No such file or directory
[...]
Reboot and ...
[...]
Why I can boot in the new kernel only one time .... ?
Click to expand...
This indeed suggests "some flaw" in the fs and my guess is that an install media booted booted fsck it will fail as well; but do try.
Also try fsck_ffs(8)

juanjico said:
[...]
Reboot and ...
[...]
Why I can boot in the new kernel only one time .... ?
Click to expand...
Before loading the kernel your screenshot indicates it cannot reliably find the UFS partition where the loader is to be loaded from "Not ufs".
However, I am puzzled as well as it seems you are able to boot once, and only once, into 14.1-RELEASE.
 
I really appreciate all the replies.

I tried the following. Starting with a running 12.4-RELEASE, I started the upgrade to 13.1-RELEASE:

1723716412112.png


Then, installed the new kernel. but instead of rebooting, I powered off the machine:

1723717166093.png


Then booted the machine with a 13.1-RELEASE (i386) boot media, go to LiveCD and run a fsck on the partition:

1723718460141.png


Seems to be clean. Tried with fsck_ffs(8) too:

1723718648796.png


Clean too. Then rebooted into the new kernel and installed userland:

1723718798909.png


Then rebooted and got the same "not ufs" problem. I can't attach the screenshot, is limited to 5.

I'm puzzled.
 
You should try to run that fsck after 2nd install, the same way you did after first one. How is your FS utilization during upgrade (is there enough free space?).
Did you try to switch to different virt console with e.g. alt+F2 to see what's happening during those busy periods (there used to be a bug with slow upgrades: here).

While you didn't share where you're running gpart bootcode during this process most likely it's not a problem not to run it at all (i.e. not related to ufs problem, at least most likely).
 
Cath O'Deray said:
At a glance: 4673370 is probably OK.
Click to expand...
Yes, but that is after first reboot, i.e. when everything is still ok after. While I also assume utilization is ok it's something to have an eye on just in case.

I also checked and confirmed you don't need to touch bootcode at all, 12.3->13.3 can be done without any bootcode upgrade (not saying it's a recommended approach, just that in this particular case it works).

Cath O'Deray said:
It's amongst the builtin commands in the loader_simp(8) manual page.
Click to expand...
Note loader is not yet loaded, this command cannot be executed at this stage with the issue he has.

To catch what is system doing during that 2nd phase having look at the 2nd console (alt-f2) would be definetely good idea. If you want to go deeper you could dd backup of the beginning of da0p2 before and after too see what has changed.
All this due to assumption you do want to know what's happening and simple reinstall/redeployment is not the point of discussion here.
As the issue is 'weird' having look at 'weird' places is not a bad idea in my opinion. Doubleckeck pmbr is OK (not having weird partition layout) to avoid possible confusion of a tool that this is MBR layout instead of GPT. While I could not think of any FreeBSD tool that would do it it's just one of those points to check. You could also update bootcode prior to the OS upgrade just to make sure this is not a problem.
 
Maybe a silly question and maybe already answered (this is a long thread), but have you checked that /boot/loader does indeed exist after you run upgrade steps?

Regarding the boot partition size, 64KB should be enough for gptboot.
If you used ZFS and needed gptzfsboot then it could be an issue.
 
You must log in or register to reply here.
Back
Top