Moving the kernel and modules after a 'nextboot -k' hides modules

[DutchDaemon]

I wonder if I should consider this a bug, a feature, or a 'duh'.

Whenever I build a new world and kernel, I routinely invoke /sbin/nextboot -k kernel.NEW/boot/kernel to make sure I can actually boot with the newly compiled kernel. If that works, I move /boot/kernel to /boot/kernel.old, and /boot/kernel.NEW/boot/kernel to /boot/. However, I do not reboot after that, I just leave everything running. It's working.

I found an interesting snag with this. I wanted to test something using amd, and it needs a couple of NFS things I do not compile into the kernel (as I don't use any NFS anywhere). Normally, the invocation of amd will simply load the NFS necessities as modules, and everything works. However, I got some interesting errors about modules that could not be found. Of course they exist, they're in /boot/kernel/, as usual.

After a bit of brain crunching, an aha moment overcame me, and I ran kldstat -v. It showed that every module that was listed there originated from /boot/kernel.NEW/boot/kernel; a location that no longer existed. Up until now, I had assumed that modules would always be searched in /boot/kernel/, whether one booted from there or not, but apparently this is not the case. And yes, a reboot was all it took.

I'm wondering whether I should make any report of this, or that this is simply a case of something that's entirely logical that I simply missed because I never ran into it. In that case I will make it my business to always reboot one more time after moving the working kernel and modules to their final location. But I do feel that /boot/kernel should always be tried, at least.

 

[kpa]

It may be that the path to the kernel file that was used for booting sets the search path for modules automatically to the containing folder.

 

[vermaden]

@DutchDaemon

I do similar, but not quite.

# cd /usr/src
# make buildworld kernel KODIR=/boot/test
# beadm create backup
# nextboot -k test

This way you have the previous kernel at /boot/kernel and the new one as /boot/test, if everything works, then you do these:

# cd /boot
# rm -rf old
# mv kernel old
# mv test kernel

Also check this: http://man.freebsd.org/kldxref

 

[DutchDaemon]

@vermaden, sure, there are ways around this, of course. But I was wondering about the general practice of setting the module search path to the one where the actual running kernel is located, and 'forgetting' the default path of /boot/kernel when calling a module.

By the way, what you describe above is exactly what I do. So you should run into the same problem when running on a kernel booted from /boot/test after moving kernel/modules over to /boot/kernel and not rebooting after that. Note: this is not about booting into that new system (because /boot/kernel and /boot/test will both exist at that point), but loading an additional module on a system that was booted from /boot/test after (re)moving that /boot/test directory.

In other words: if kldstat -v shows your module directory path as /boot/test, you won't be able to load a new module after removing /boot/test. And I'm saying: would be nice if /boot/kernel was at least attempted.

 

[vermaden]

In other words: if kldstat -v shows your module directory path as /boot/test, you won't be able to load a new module after removing /boot/test. And I'm saying: would be nice if /boot/kernel was at least attempted.

kldxref does not fix that?

 

[DutchDaemon]

Like I said, there are ways around it, it's just the default behavior (i.e. not trying /boot/kernel and /boot/modules when not being able to find a module) that I question

 

[kpa]

I think this would be an easy fix. Add the custom module paths to the kern.module_path loader(8) variable before the standard paths /boot/kernel;/boot/module instead of replacing them. It's all set in the loader and what nextboot(8) does is just add a file with extra settings that are read at boot.

 

[DutchDaemon]

Ok, I must have formulated the issue wrong, because people keep suggesting fixes to me that I know, and therefore don't need This was a matter of 'what the default system behavior should be', not 'how do I load my modules?'

I'll mark this as solved, as in 'the answer to the first line and the final paragraph of the original post is "whatever"'.

 

[phoenix]

This is a "duh!" moment, not a bug or a feature.

When you use nextboot(8), the loader uses that to set the module path that gets sent to the kernel, and which the kernel uses to search for modules. IOW, it's working as designed... just not how you want.

And, if you think about it, you really don't want it to work any other way. You used nextboot for a reason (to load the kernel from a non-standard location, to give you a fall-back to the standard kernel if things go sideways). And you really don't want this non-standard kernel to be pulling modules from the standard locations as that would mix modules from separate kernels... a definite no-no.

The kernel has no way of knowing you changed directories out from underneath it. However, I believe you can tell the kernel that you changed the directories underneath it by editing the module_path kenv variable or via sysctl. Never tried it, though.

Ideally, you should add "reboot" into your testing framework, as the step right after "move test kernel to /boot/kernel". After all, that's how I do things, and I'm obviously the greatest FreeBSD user ever.

 

[DutchDaemon]

Yeah, I can suffer the extra reboot, as I'm not rebooting between the make installkernel and make installworld steps anyway