Solved 13.0-p4 to 13.0-p6 module oddness

M

mer

Morning all. Finally got around to freebsd-update one of my systems to -p6. Rebooting into the p6 kernel was "interesting". Problem boiled down to "what modules are loaded" after/during boot. Yes there is an NVIDIA card in use, USB keyboard and mouse.
Any thoughts? I have done a ton of investigation yet (trying to get paying work done). Almost "feels" like after doing the update that cross refs or something for the modules is mucked up and kernel is not picking it up correctly (kldxref I think).

First output of kldstat from the -p4 boot
Code: 
 cat kldstat-p4.txt
Id Refs Address                Size Name
 1   55 0xffffffff80200000  1f11f28 kernel
 2    1 0xffffffff82112000     b7b8 opensolaris.ko
 3    1 0xffffffff8211e000   67feb0 zfs.ko
 4    1 0xffffffff82918000     3250 ichsmb.ko
 5    1 0xffffffff8291c000     2180 smbus.ko
 6    1 0xffffffff8291f000   107310 nvidia-modeset.ko
 7    1 0xffffffff82c00000  1faac70 nvidia.ko
 8    2 0xffffffff82a27000    388f8 linux.ko
 9    3 0xffffffff82a60000     db70 linux_common.ko
10    1 0xffffffff82a6e000     20f0 coretemp.ko
11    1 0xffffffff82a71000     2340 uhid.ko
12    1 0xffffffff82a74000     4350 ums.ko
13    1 0xffffffff82a79000     3380 usbhid.ko
14    1 0xffffffff82a7d000     31f8 hidbus.ko
15    1 0xffffffff82a81000     3320 wmt.ko
16    1 0xffffffff82a85000    27040 ipfw.ko

now kldstat in the p6 after I have hand loaded a couple of modules: linux_common, linux, nvidia-modeset and nvidia just to prove they could be:
Code: 
cat kldstat-p6.txt
Id Refs Address                Size Name
 1   26 0xffffffff80200000  1f11f90 kernel
 2    1 0xffffffff82112000     b7b8 opensolaris.ko
 3    1 0xffffffff8211e000   67feb0 zfs.ko
 5    1 0xffffffff82918000     20f0 coretemp.ko
 6    4 0xffffffff8291b000     db70 linux_common.ko
 7    3 0xffffffff82929000    388f8 linux.ko
 8    1 0xffffffff82962000   107310 nvidia-modeset.ko
 9    1 0xffffffff82c00000  1faac70 nvidia.ko
Same ZFS BE, same init scripts, literally nothing but freebsd-update fetch && freebsd-update install && shutdown -r now.

/boot/loader.conf:
Code: 
cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
opensolaris_load="YES"
zfs_load="YES"
hw.vga.textmode="1"
vfs.zfs.arc_max="8589934592"
#hw.usb.no_boot_wait="1"
hw.usb.no_shutdown_wait="1"

Nothing jumping out in dmesg between good and bad. So any ideas/thoughts/"you idiot you forgot to ...." are welcome.

Oh,
Thanks
 
mer said:
2 1 0xffffffff82112000 b7b8 opensolaris.ko
Click to expand...
Why are you loading opensolaris.ko? OpenZFS on 13.0 doesn't need it any more. Pre-13.0 ZFS did require it but would automatically load it as a dependency of zfs.ko. There was never a need to explicitly load opensolaris.ko (only if you dropped to the loader(8) prompt and manually loaded the kernel and modules).
 
Good question; answer is "probably because it's been there since FreeBSD-9.0?" so I'll clean that up.
But I don't think that is a root cause of the issue.
thanks

ETA:
So just modified loader.conf to not load it, manually did kldunload and confirm that 13.0 does not need it.
 
What exactly is the problem? You only mention that it's "interesting", without actually explaining what was interesting about it.
 
Ok, problem statement:
Ran freebsd-update to pull latest updates, rebooted into p6 kernel and system was unusable.

Investigating "why" I realized that a lot of the kernel modules that were automatically loaded in previous kernel (FreeBSD-RELEASE-13.0-p4) were not being loaded in the -p6 update.
Booting into the p6 and manually loading the module created a system that was useable.

The interesting part is:
Why did the modules not load automatically? They could be loaded by hand without error. None of the initialization bits (loader.conf, init scripts) changed between the updates.
Notice that modules related to usb mouse and keyboard did not load uhid, ums and friends.
Stopping at loader prompt and doing the unload; load kernel.old, zfs.ko boot worked (booting back into -p4).

Hopefully that is clearer as to a problem and interesting information.
 
mer said:
Investigating "why" I realized that a lot of the kernel modules that were automatically loaded in previous kernel (FreeBSD-RELEASE-13.0-p4) were not being loaded in the -p6 update.
Booting into the p6 and manually loading the module created a system that was useable.
Click to expand...
I have a whole bunch of systems I updated from p4 to p6 last week, none had issues (almost all of them are running on ZFS). So it's not a "common" problem with some bad linkage with the update files from freebsd-update(8). I would have expected a LOT of noise about it by now if there was indeed a problem with it.

mer said:
Why did the modules not load automatically? They could be loaded by hand without error.
Click to expand...
If there was some "disconnect" between the kernel and the modules then loading them by hand would fail too.

Have you tried a verbose boot? That might provide some clues.
 
  • Like
Reactions: mer
Did I get it correctly then you're asking why you don't have the same list of the modules after reboot to p6 as you had on p4 ? For some (e.g. wmt, ums) it could be they were loaded automatically when device required it or when program that uses it loaded it (maybe something X related).

Interesting one is ipfw. Do you use it as primary FW ? Maybe it's also leftover ..
 
Your first sentence is more or less correct.
I understand about the dynamic loading, but nothing was changed in init services between the 2 kernels, so moused should be loading ums.ko and it's dependencies, etc.


Yes, IPW is used on the system in question as a firewalling for a "workstation", following lines from /etc/rc.conf.
firewall_enable="YES" firewall_type="workstation"

obviously not a common thing for the upgrade, just kind of posting to see if anyone has any ideas as to potentially "why".
I guess you can say "if we were in the office, this is me talking out loud over the cubicle walls, looking for any straw that anyone may have".
 
I guess we can mark this solved.
I added a line to my /etc/rc.conf:

kldxref_clobber="YES"

so that linker.hints is regenerated every boot.
A bit of searching wound up on a thread from back in FreeBSD-7.x days that talked about this. Not sure if the default value changed between -p4 and -p6 but that's what seems to have fixed it.
 
You must log in or register to reply here.
Back
Top