Upgrading from 14.3-p6 to 15.0: ipfw_ctl3 invalid option 98v0, 97v0

[openglx]

Hi all,

I had some minor trouble with the ipfw blocking all connections when I upgraded from FreeBSD 14.3-RELEASE-p6 to FreeBSD 15.0-RELEASE on my Intel NUC home server.
My installation is very vanilla: originally installed from 14.1, upgrading with freebsd-update, ZFS root, thick jails with vnet, no pkgbase.

Before upgrading I've read the release notes and the upgrade documentation at https://www.freebsd.org/releases/15.0R/installation/#upgrade-binary and found nothing particularly worrying for my setup.

I've confirmed I was on 14.3-p6 before upgrading, running freebsd-update fetch install said no updates pending. It's possible I wasn't on the latest kernel but certainly on latest userland.
I had no errors on freebsd-update upgrade -r 15.0-RELEASE or freebsd-update install.
After reboot, entered GELI password, boot screen as normal, system loading as normal.
I started seeing several errors on console

ipfw: ipfw_ctl3 invalid option 98v0

The code

98v0

sometimes changed to

97v0

or

99v0

.

The computer was unable to reach any host on the network, even ping to its gateway would return

ping: sendto: Permission denied

.

Trying to check what could be wrong, I noticed that ipfw list would error with:

ipfw: retrieving config failed: Invalid argument

and the console would log

ipfw: ipfw_ctl3 invalid option 97v0

I knew I was in uncharted territory running a 15.0 kernel but 14.3 userland.
I completed the upgrade with freebsd-update install and rebooted once more.
The problem when away on the host system but I still see this error happening on any vnet jail.
Fortunately the fix appears to be as simple as upgrading each jail and restarting them.

Could you please advise what have I done wrong, or if there'a any additional information I could present to help anyone else in the future?

Thanks for the great work and congratulations on the successful release.

 

[AndAlooongCameFreeBSD]

Had the same issue, I doubt that you did something wrong. When things like this happen I grep through the source code, trying to find the error string. So the error comes from /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c and without having really understood what I found in there it appears to me that somehow an integer value indicating IPFW's version does not have the value it should have at the time you and I saw that message. Not sure what makes the error disappear, I did ipfw disable firewall to get pkg working, updated all packages, did the last freebsd-update install and rebooted.

Just keep in mind that you should pkg update && pkg upgrade after it says something along the lines that 3rd party packages from ports need to be reinstalled and before the following (usually the last) freebsd-update install. Otherwise something important might stop working because some shared library is not where it used to be.

 

[openglx]

Thanks for the reply, AndAlooongCameFreeBSD.
I appreciate the suggestion of trying to find the culprit.

I noticed a post appeared on r/freebsd a few days after I originally posted here: https://www.reddit.com/r/freebsd/comments/1pmgu39/remote_update_to_freebsd_15_failed_because_of/

The top reply there was by "NickBergenCompQues"t:

This a catch 22 with the FreeBSD 15.0 upgrade, introduced a ABI change between ipfw(8) and the kernel ipfw(4).

So after the first reboot you have a new kernel but still have the old ipfw(8). So they can’t talk to each other, and the old ipfw can’t program the new kernel. The default rule is deny all when no rules are set, so SSH gets blocked.

So I would:
    Disable ipfw before for the first reboot
    Do the upgrade install again to update the userland
    Reboot

    Then turn ipfw back on to see if it can work with your new system

—————————————————————

Hope this helps. Let me know if it works. (Maybe there’s something else introduced into the mix and I’m wrong.)

It was suggested that this information about the ipfw ABI change should've been flagged on the release docs.
I think it would've helped to respect the Principle of Least Astonishment.

On the long schema of things, it was a pretty mild inconvenience and I was prepared to field much worse problems on a major upgrade.
Fortunately, it took me seconds to solve it.

Kudos for the release team for a great upgrade experience overall.

Thanks.