Solved pkg refuses to upgrade firmware packages to 15 version

cadahl

cadahl

After upgrading 14.3 to 15 using freebsd-update, my system refused to boot due to a panic when loading the amdgpu driver.

After investigating and discussing it with people on the discord, I noticed that none of the firmware packages had been upgraded properly. They still have the 140300 version suffix.

pkg upgrade -f does nothing at this point, but if I remove and then install a specific firmware package, the correct version is installed.

Doing this for the AMD GPU firmware of the flavor needed solves the kernel panic at boot.

Example with a different firmware package, same symptoms showing the problem:

pastebin.com

# uname -aFreeBSD m715q 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995- - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com
 
cadahl said:
I noticed that none of the firmware packages had been upgraded properly. They still have the 140300 version suffix.
Click to expand...
I think they don't have to be in sync with the kernel like kernel modules. They are just firmwares with no special link with the host OS. Are your kmods in sync with the kernel? That's really important.

Looking with pkg search, it appears that gpu-firmwares are tagged 1500068 on the latest repo.

Try to pkg delete kmod related packages and reinstall them. I do this systematically for OS upgrade with the FreeBSD-ports-kmods repos, because pkg upgrade tends to not upgrade these specific packages.
 
Thanks, I've solved it locally as described in the post, but figured I'd let others know in case someone else runs into the issue. And just in case it was an actual bug with pkg.
 
I wonder if I have not well read your post or if you have edited it. In all cases, I doubt that the firmware is so important. I repeat, it's just code that is not compiled under FreeBSD but just copied from upstream.

I saw that deinstalling a firmware also removing drm-kmod. Wasn't that the real problem?
You needed to reinstall it?
 
Emrion said:
I wonder if I have not well read your post or if you have edited it. In all cases, I doubt that the firmware is so important. I repeat, it's just code that is not compiled under FreeBSD but just copied from upstream.

I saw that deinstalling a firmware also removing drm-kmod. Wasn't that the real problem?
You needed to reinstall it?
Click to expand...
I thought I was fairly clear: the system wouldn't boot, due to an error in the amdgpu driver. After replacing the firmware package with the version ending in 1500068, the system booted.

I have verified this by going back and forth between the different firmware packages. I don't know what the actual difference is.

I have since making the post replaced all the packages that were wrong.
 
cadahl said:
I thought I was fairly clear: the system wouldn't boot, due to an error in the amdgpu driver. After replacing the firmware package with the version ending in 1500068, the system booted.

I have verified this by going back and forth between the different firmware packages. I don't know what the actual difference is.

I have since making the post replaced all the packages that were wrong.
Click to expand...
Hey, thank you, this helps me as well. I have upgraded and rolled back twice trying different methods. I didn't think to check the version of driver being installed. 👍
 
I didn't see that sort of firmwares contain kmods. I'm yet pretty sure I left them as this when you needed to recompile the drivers at each minor version. That said, the firmwares versions are the same between 14.3 and 15.0. The problem is maybe because it's a major upgrade.

So, I think that each kmod is responsible for loading the firmware (*.bin) into the hardware with the help of the kernel API. So, it maybe incompatible from different major versions. It makes sense.
 
I'm also getting a kernel panic loading amdgpu at boot time.

I also had 14.3 references in some of the installed packages. I found that if I deleted a package and reinstalled it, the install brought in the 15.0 version of the package.
So I suppose I need to find all packages with old 14.3 references and delete them and reinstall them to get the new 15.0 versions of them all.
When the package is reinstalled it brings in the kernel .ko files too I think?

Here is my kernel panic log below - note the 'not available or version mismatch' references towards the end of the log.

I will read about how it was fixed above. Thanks!

Code: 
KERNEL PANIC LOG
----------------
<118>Loading kernel modules: amdgpu
<6>[drm] amdgpu kernel modesetting enabled.
drmn0: <drmn> on vgapci0
vgapci0: child drmn0 requested pci_enable_io
vgapci0: child drmn0 requested pci_enable_io
<6>[drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x164E 0x1462:0x7D77 0xC4).
<6>[drm] register mmio base: 0xF6600000
<6>[drm] register mmio size: 524288
<6>[drm] add ip block number 0 <nv_common>
<6>[drm] add ip block number 1 <gmc_v10_0>
<6>[drm] add ip block number 2 <navi10_ih>
<6>[drm] add ip block number 3 <psp>
<6>[drm] add ip block number 4 <smu>
<6>[drm] add ip block number 5 <dm>
<6>[drm] add ip block number 6 <gfx_v10_0>
<6>[drm] add ip block number 7 <sdma_v5_2>
<6>[drm] add ip block number 8 <vcn_v3_0>
<6>[drm] add ip block number 9 <jpeg_v3_0>
drmn0: Fetched VBIOS from VFCT
<6>amdgpu: ATOM BIOS: 102-RAPHAEL-008

amdgpu/psp_13_0_5_toc.bin: could not load binary firmware /boot/firmware/amdgpu/psp_13_0_5_toc.bin either
psp_13_0_5_toc.bin: could not load binary firmware /boot/firmware/psp_13_0_5_toc.bin either
amdgpu_psp_13_0_5_toc.bin: could not load binary firmware /boot/firmware/amdgpu_psp_13_0_5_toc.bin either
KLD amdgpu_psp_13_0_5_toc_bin.ko: depends on kernel - not available or version mismatch
amdgpu_psp_13_0_5_toc_bin: could not load binary firmware /boot/firmware/amdgpu_psp_13_0_5_toc_bin either
KLD amdgpu_psp_13_0_5_toc_bin.ko: depends on kernel - not available or version mismatch
amdgpu_psp_13_0_5_toc_bin: could not load binary firmware /boot/firmware/amdgpu_psp_13_0_5_toc_bin either
drmn0: could not load firmware image 'amdgpu/psp_13_0_5_toc.bin'
[drm ERROR :amdgpu_device_ip_early_init] early_init of IP block <psp> failed -19
 
froggit9000 said:
So I suppose I need to find all packages with old 14.3 references and delete them and reinstall them to get the new 15.0 versions of them all.
Click to expand...
As in the handbook wrote you've got to reinstall all packages after a major version update (wasn't necessary on all major upgrades so far, but on this on it is). These days I found a lot of people in this forum just doing a "pkg upgrade" - that's not enough an will lead to problems. One of these:
pkg-static upgrade -f
pkg upgrade -f

I'm also cleaning the package cache prior to this ("pkg clean -a"), but that shouldn't be necessary. So no, you don't have to find 14.3 packages as none of it should remain anyway.
 
jmos said:
As in the handbook wrote you've got to reinstall all packages after a major version update (wasn't necessary on all major upgrades so far, but on this on it is). These days I found a lot of people in this forum just doing a "pkg upgrade" - that's not enough an will lead to problems. One of these:
pkg-static upgrade -f
pkg upgrade -f

I'm also cleaning the package cache prior to this ("pkg clean -a"), but that shouldn't be necessary. So no, you don't have to find 14.3 packages as none of it should remain anyway.
Click to expand...
pkg upgrade -f did not work.

I had to remove, then install the packages again.
 
…you're right:
root@rosmerta:~ # pkg info
gpu-firmware-amd-kmod-picasso-20230625.1403000_2 Firmware modules for picasso AMD GPUs
pkg-2.4.2 Package manager
So I've got only 2 packages installed: pkg itself an a kmod package.
root@rosmerta:~ # pkg upgrade -f
[…]
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
pkg-2.4.2 [FreeBSD-ports]

Number of packages to be reinstalled: 1
As the manpage of pkg-upgrade explains a pkg upgrade -f "Force the reinstallation or upgrade of the whole set of packages." that's not true in case of my kmod test (gone the whole way from upgrading a basic 14.3 machine). IMO you should write a bug report…
 
jmos said:
…you're right:
root@rosmerta:~ # pkg info
gpu-firmware-amd-kmod-picasso-20230625.1403000_2 Firmware modules for picasso AMD GPUs
pkg-2.4.2 Package manager
So I've got only 2 packages installed: pkg itself an a kmod package.
root@rosmerta:~ # pkg upgrade -f
[…]
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
pkg-2.4.2 [FreeBSD-ports]

Number of packages to be reinstalled: 1
As the manpage of pkg-upgrade explains a pkg upgrade -f "Force the reinstallation or upgrade of the whole set of packages." that's not true in case of my kmod test (gone the whole way from upgrading a basic 14.3 machine). IMO you should write a bug report…
Click to expand...
To find if there are other 14.3 packages remaining you can try:

Code: 
# pkg info -a | grep 1403000

The output is like:

Code: 
$ pkg info -a | grep 1403000
gpu-firmware-amd-kmod-arcturus-20230625.1403000_2 Firmware modules for arcturus AMD GPUs
gpu-firmware-amd-kmod-banks-20230625.1403000_2 Firmware modules for banks AMD GPUs
gpu-firmware-amd-kmod-beige-goby-20230625.1403000_2 Firmware modules for beige_goby AMD GPUs
gpu-firmware-amd-kmod-bonaire-20230625.1403000_2 Firmware modules for bonaire AMD GPUs
gpu-firmware-amd-kmod-carrizo-20230625.1403000_2 Firmware modules for carrizo AMD GPUs
gpu-firmware-amd-kmod-cyan-skillfish2-20230625.1403000_2 Firmware modules for cyan_skillfish2 AMD GPUs
gpu-firmware-amd-kmod-dimgrey-cavefish-20230625.1403000_2 Firmware modules for dimgrey_cavefish AMD GPUs
gpu-firmware-amd-kmod-fiji-20230625.1403000_2 Firmware modules for fiji AMD GPUs
gpu-firmware-amd-kmod-green-sardine-20230625.1403000_2 Firmware modules for green_sardine AMD GPUs

We could make a little script that takes each line of the output...
e.g. "gpu-firmware-amd-kmod-arcturus-20230625.1403000_2 Firmware modules for arcturus AMD GPUs"
and strips the "-20230625.1403000_2 Firmware modules for arcturus AMD GPUs" leaving only the package name.
In this example it would give us "gpu-firmware-amd-kmod-arcturus" which the script could then "pkg delete" and then "pkg install".

I did this manually for 32 packages and it's tedious.

Is anyone good at writing shell scripts? I'm not :(

Script outline:
Code: 
packages = pkg info gpu-firmware-amd-kmod-* | grep 1403000
while (package) {
  package_name = accept all characters until first numerical character found ('0-9'), then remove trailing '-' char
  pkg delete package_name
  pkg install package_name
}
 
Last edited:
Something like this perhaps? Disclaimer: it's probably a ham-fisted attempt at a script, but it should work. It splits the package name at the last dash character, which should be more appropriate. I still need to learn what the rules are for FreeBSD package names.

Code: 
#!/bin/sh
PKGS=$(pkg info | grep -o '^[^ ]*1403000[^ ]*')

for PKG in $PKGS; do
  VERSIONSUFFIX=$(echo $PKG | awk -F '-' '{ print $NF }')
  PKG_NAME=$(echo $PKG | sed "s/-$VERSIONSUFFIX//")

  # remove the echos once you've double-checked that the output makes sense...
  echo pkg delete -y $PKG_NAME
  echo pkg install -y $PKG_NAME
done
 
You can, but you don't need* to use a 'programmed' script:
  1. select package names (version numbers contain 1403000) and write to a file for manual inspection as desired.
  2. feed that file to pkg-install(8), using -f does a forced reinstallation.

Edit: **
Rich (BB code): 
# pkg query '%n %v' | grep 1403000 | cut -d ' ' -f 1 > pkg.list
# pkg delete -f `cat pkg.list`
# pkg install -f `cat pkg.list`
In sh(1), instead of using backtick quotes, the new "superior style" of command substitution is:
pkg install -f $( cat pkg.list )

___
* I've found these very useful:
  1. Sh - the POSIX Shell
  2. Regular Expressions
  3. Grep - An introduction to grep and egrep.
  4. Sed - An Introduction and Tutorial by Bruce Barnett
  5. AWK
Allthough, of these, you only need a bit of #1, as the regex 1403000 is very basic.

** EDIT: # pkg delete -f `cat pkg.list` was added after the correct observation of cadahl ; see below
 
Last edited:
I don't think -f works, as previously mentioned. PKG does not consider the packages related (?) so does not upgrade or reinstall. They had to be first deleted, then installed again by name.

That's the reason for this thread in the first place.
 
You must log in or register to reply here.
Back
Top