Solved Another 13.1-RELEASE failure to boot

Greetings all,

I decided to upgrade from 12.3-RELEASE to 13.1-RELEASE, which is allegedly supported, since I have a GENERIC kernel.

Since I am paranoid, I first executed:
# freebsd-update fetch
# freebsd-update install
,
and rebooted the machine:
# shutdown -r now.

Then I executed:
# freebsd-update upgrade -r 13.1-RELEASE
# freebsd-update install


After merging about several files, I rebooted the machine again: # shutdown -r now, and was rewarding by:
Code:
. . .
loading required module 'kernel'
module 'kernel' exists but with wrong version
at which the machine was sitting like a dead duck.

I did not panic yet, as I initialized BE, and by restarting the machine, and selecting Option 8 "Boot Environments"
Code:
1. Back to main menu [Backspace]     
2. Active: zfs:system/BE/default (1 of 3)     
3. bootfs: zfs:system/BE/default
and further 2, I saw the "default" being "12.3-RELEASE". The machine appears to go through a boot sequence, but ends with a message:
Code:
Starting background file checks in 60 seconds.
However, it has been presenting the message for about an hour, so I do not think that this is a solution.

Option 3 then ends with the same message
Code:
. . .
loading required module 'kernel'
module 'kernel' exists but with wrong version

As the last resort, I loaded Option
Code:
6 Kernel: kernel.old (2 of 2),
, which was successful.

So, I am not sure, what to do now. From what I understand, since the BE did not work as advertised, I am running the old kernel with some of the different configuration files - due to the merge, which does not seem like a good thing. Can I somehow get to the original 12.3 configuration? Should I re-attempt the upgrade again? Should I just re-install?

Any help would be appreciated.

Kindest regards,

M
 

SirDice

Administrator
Staff member
Administrator
Moderator
I am running the old kernel with some of the different configuration files - due to the merge
Those merges happen before installation. They haven't been installed yet. The fist freebsd-update install only copies the new kernel, nothing else. It's the second freebsd-update install that will update your userland (including the merged configuration files).

Do you have any kernel modules loaded (check loader.conf)? Specifically DRM or Virtualbox modules? Best to temporarily disable them during the upgrade process. When the kernel is upgraded those modules are still for 12.x, and will fail to load on a 13.x kernel. You can enable them again after you completed the entire upgrade procedure.
 
Hi SirDice,

thank you for the explanation. I have misread the instructions https://www.freebsd.org/releases/13.1R/installation/. Since the instructions mention merging configuration files before the first freebsd-update install command, I (erroneously) concluded that the configuration files are updated together with the (new) kernel install.

I do not have any kernel modules loaded by the /boot/loader.conf.

Apart form the above, why did the BE not work?

Kindest regards,

M
 

SirDice

Administrator
Staff member
Administrator
Moderator
I do not have any kernel modules loaded by the /boot/loader.conf.
kld_list in rc.conf perhaps? Disable those, just for the upgrade. Basically, disable any and all services, kernel modules, etc. that aren't strictly necessary for the system to boot (or to do remote management; so you may want to leave sshd_enable). Bring it down to the bare minimum. You can enable all those again when you've finished with the upgrade.

Apart form the above, why did the BE not work?
freebsd-update(8) can create a BE before the actual upgrade. It's not enabled by default though. You may have created those by hand in the past, maybe they weren't correct to begin with.

Code:
# Create a new boot environment when installing patches
# CreateBootEnv yes
 
Hi SirDice,

thank you for your reply again.

Regarding your advice, I did some reading in a meanwhile and found that some people needed to disable
Code:
nvidia_load="YES
in the /boot/loader.conf. So did I, and repeated the upgrade. It appeared to work, since after restart I confirmed by uname -a that the new kernel was installed, and I was asked to rebuild all the third party software, which I am currently doing.

I will not mark the thread solved, until after it will have finished and the second third freebsd-update install is successfully executed, not to jinx the current progress. ;)

Regarding the BE, I did create them when I originally installed 12.1-RELEASE and have been using them to upgrade all the way to 12.3-RELEASE. As such the BE have been working. Hence my surprise that they did not work after the failed upgrade.

Kindest regards,

M
 
From the man pages*:
The option suggested by grahamperrin in his message:
  • to not create a boot environment, CreateBootEnv no
seems appropriate when you do not want BEs to be created automatically.

For manually creating BEs, this may help:

___
* compare FreeBSD’s 12.3-RELEASE freebsd-update.conf(5) with FreeBSD’s 13.1-RELEASE freebsd-update.conf(5); the latter has the option CreateBootEnv.
 

SirDice

Administrator
Staff member
Administrator
Moderator
and I was asked to rebuild all the third party software, which I am currently doing.

until after it will have finished and the second freebsd-update install is successfully executed
Hang on. You're supposed to (re)install all your third party software (aka ports/packages) after the second install run, not after the first. You need to do this after the second install because the second install installs the new libraries and other userland tools. It's those you're building for.

With a major version upgrade; First freebsd-update install only installs the new kernel. You should reboot but honestly I rarely do. Then run the second freebsd-update install. This updates libraries, userland, everything else except the kernel (that was already done in the first). At this point should you rebuild your ports (or reinstall your packages). Once all that is done, a third freebsd-update install will remove all of the old libraries, tools and whatever else needs removing.
 
[...] I confirmed by uname -a that the new kernel was installed [...]
To get a good look at what's going on with the installed kernel, running kernel and userland with the three times you issue freebsd-update install, execute freebsd-version -kru after each reboot (instead of uname). With every step in the upgrade process you'll see the progress.
 
Hi SirDice,

sorry, I made a clerical error, the promt to re-install the third party applications came after the second freebsd-update install. I will correct my previous post.

Hi Erichans,

could you please clarify your post re BE? As I wrote, I did create (manually) the BE during installation of 12.1. Hence my surprise that they did not work, since I clearly saw and selected in the "Options" the BE pointing to 12.3.

Thank you for the freebsd-version -kru. Will use it.

Kindest regards,

M
 
[...] I did not panic yet, as I initialized BE, and by restarting the machine, and selecting Option 8 "Boot Environments"
[...] could you please clarify your post re BE? As I wrote, I did create (manually) the BE during installation of 12.1. Hence my surprise that they did not work, since I clearly saw and selected in the "Options" the BE pointing to 12.3.
I may be interpreting this completely wrong but, If I take the first quote literally then you started creating a BE during the upgrade process: "I did not panic yet, as I initialized BE" looks to me like "I did not panic yet, and created a BE". If that's the case then you could have been returned to a BE that was created whilst being in the process of upgrading from 12.3-RELEASE to 13.1-RELEASE.

Taking the second quote, looks like you created a BE of 12.1: that is quite an old BE if you started this upgrade from 12.3-RELEASE. The idea in such a major upgrade (from one major FreeBSD version to the next) is that you create a BE before (just before, or at least very recently before) you start the upgrade.

A good idea is to be sure that you are on the most recent patch level of the 12.3-RELEASE; then create a BE with an appropriate name. Then proceed with the first step in the upgrade process. Again, I may be misinterpreting your messages. Apart from that, without a sequence of events (like this for example) it is hard to know what happened exactly and where things might have gone wrong.

I'm presuming that you did not "discard" any BE? Either by bectl(8) or beadm(8). (I've found the Klara article helpful.) Please look at the output of bectl list -c creation and see what the date-timestamp is of the BE to which you have returned.
 
Hi Erichans,

thank you for the reply.

The confusion was caused by my in-artful use of English; I should have written "I did not panic yet, as I had initialized BE".

And, yes, I had created BE upon installation of 12.1 and a new BE for each upgrade from 12.1 to 12.3. And, no, I did not discard any of the BEs, hence my surprise that the one marked 12.3 did not work.

However, it appears that I am on the right path now; there is still one port to be compiled and then I try your suggestion freebsd-version -kru.

Kindest regards,

M
 
Hi SirDice, Erichans,

thank you very much for your help.

Unfortunately, after I updated all the third party applications and confirmed with the freebsd-version -kru that I am on 13.1-RELEASE, the next reboot ended with the, now familiar, message:
Code:
. . .
loading required module 'kernel'
module 'kernel' exists but with wrong version

The system now shows four BEs, two for the previous 12.3-RELEASE, two for the 13.1-RELEASE, the latter two dated yesterday and today. Interestingly, the above message is produced by using today's 13.1-RELEASE date BE, yesterday's 13.1-RELEASE date BE executes until the, also familiar, message:
Code:
Starting background file checks in 60 seconds.
, after which the system is non-responsive.

Similarly, none of the 12.3-RELEASE BEs work.

In conclusion, after wasting two days, I am worse off than I was before.

Kindest regards,

M
 
Greetings all,

I made some progres by selecting Option 3, and in the loader prompt executing:
Code:
OK unload
OK boot /boot/kernel
I then commented out
Code:
nvidia-load="YES"
from /boot/loader.conf, upon which the system flawlessly booted. I noticed that the nvidia driver was updated, so I reinstalled the driver and all now works.

It is rather confusing that the system complained about a problem with a kernel, since a few lines above it reported that nvidia.ko was loaded. I would have never consider this to be a problem without SirDice's advice.

I also do not understand, why the driver was no updated unlike the other applications.

Kindest regards,

M
 

SirDice

Administrator
Staff member
Administrator
Moderator
I also do not understand, why the driver was no updated unlike the other applications.
It should be updated after the second freebsd-update install, when you're rebuilding (or reinstalling) all your ports/packages. It's not part of the base OS so it doesn't get updated with freebsd-update(8).
 
[...] I noticed that the nvidia driver was updated, so I reinstalled the driver and all now works.

It is rather confusing that the system complained about a problem with a kernel, since a few lines above it reported that nvidia.ko was loaded. I would have never consider this to be a problem without SirDice's advice.

I also do not understand, why the driver was no updated unlike the other applications.

On 12.3-RELEASE, I have:
Code:
% kldstat
Id Refs Address                Size Name
 1   39 0xffffffff80200000  2295a98 kernel
 2    1 0xffffffff82496000   da8e80 nvidia.ko
<snap>
You have:
[...] Unfortunately, after I updated all the third party applications and confirmed with the freebsd-version -kru that I am on 13.1-RELEASE, the next reboot ended with the, now familiar, message:
Code:
loading required module 'kernel'
module 'kernel' exists but with wrong version
My observation is that nvidia.ko (that is a kernel module) is trying to load the module 'kernel' from your previous major version; that is the kernel of the OS of the 12.3-RELEASE. That version of the kernel (module) is still present—FreeBSD keeps an old kernel around*. It cannot load this old kernel version because it clashes with the one already loaded: the first entry in the output of kldstat; the one already loaded is the 13.1-RELEASE kernel module in your case.

As SirDice mentioned, it is important that you update your nvidia driver at the right moment. Either by pkg or by building it from source. If you update your nvidia driver when the running kernel of the old version (12.3-RELEASE) is still active, you are getting/building the driver not targeted for your 13.1-RELEASE.

___
* changed: only keeps one old kernel in /boot/kernel.old
 
Last edited:
Hi SirDice,

It's not part of the base OS so it doesn't get updated with freebsd-update(8).
I understand that. Since I have a few ports where I use non-default option, I have used synth(1) to rebuild the non-base ports. However, being a moron, I forgot to refresh the /port/usr. :oops: So, it makes sense now.

Hi Erichans,

My observation is that nvidia.ko (that is a kernel module) is trying to load the module 'kernel' from your previous major version; that is the kernel of the OS of the 12.3-RELEASE. That version of the kernel (module) is still present—FreeBSD keeps a number of old kernels around in its base install (unless you build your own base install from source and specify otherwise).

I know that the old kernel is present; I tried to load it from the loader prompt OK /boot/kernel.old, but for some reason it did not work, though I have confirmed that the kernel.old does exist in /boot.

Regarding your first sentence, it seems to make perfect sense based on what I observed. It again shows my lack of understanding, which was that the /boot/kernel is loaded first and then the kernel modules afterwards. So, thank you for the explanation.

I want to retract my comment about "wasting two days" because I have learnt something so although I spent two days on the problem, they were not wasted.

Kindest regards,

M
 
I know that the old kernel is present; I tried to load it from the loader prompt OK /boot/kernel.old, but for some reason it did not work, though I have confirmed that the kernel.old does exist in /boot.
As both /boot/kernel and /boot/kernel.old are directories, from the loader prompt try:
Code:
OK load /boot/kernel.old/kernel
 

SirDice

Administrator
Staff member
Administrator
Moderator
Also make sure to load zfs.ko from that same kernel.old directory if you're booting from ZFS. If it's ZFS from < 13.0, you need to load opensolaris.ko first, then the zfs.ko module.
 
Hi Erichans,

I will try the command that you recommended out of curiosity, thank you.

Hi SirDice,

I looked at the /boot/loader.conf of the 12.x-RELEASE, but there is no directive to load opensolaris.ko, yet it has worked from 12.1 to 13.3.

Kindest regards,

M
 

SirDice

Administrator
Staff member
Administrator
Moderator
I looked at the /boot/loader.conf of the 12.x-RELEASE, but there is no directive to load opensolaris.ko, yet it has worked from 12.1 to 13.3.
It's normally automatically loaded as a dependency of zfs.ko. But from the loader prompt there's no automatic loading, so you have to do this by hand. On 13.0 and higher you don't need opensolaris.ko anymore because 13.x switched to OpenZFS.
 
Hi SirDice,

since we had, more or less, moved from the original problem, can you please clarify? If the opensolaris.ko is
normally automatically loaded as a dependency of zfs.ko.
why are you saying that
But from the loader prompt there's no automatic loading, so you have to do this by hand.
My 12.X /boot/loader.conf has a line
Code:
zfs_load="YES"
, which, as best as I understand loads the zfs.ko, which then loads opensolaris.ko. In 13.X, the
Code:
zfs_load="YES"
is replaced with
Code:
openzfs_load="YES"
Cf, e.g., https://openzfs.github.io/openzfs-docs/Getting Started/FreeBSD.html.

Or am I still missing something?

Kindest regards,

M
 
Hi SirDice,

I actually installed 13.1-RELEASE on another computer and found the
Code:
zfs_load="YES"
in the /boot/loader.conf.

So, thank you for the explanation.

Kindest regards,

M
 
Top