Solved Upgrading FreeBSD 12.2 to 13.0 via NFS

Hello, I'm trying to upgrade a vm from 12.2 to 13.0 via NFS and I receive the following error message:

Code:
root@fbtest:/usr/src # make installkernel

--------------------------------------------------------------

>>> Install check kernel

--------------------------------------------------------------

--------------------------------------------------------------

>>> Installing kernel GENERIC on Sun May 16 08:36:45 EEST 2021

--------------------------------------------------------------

cd /usr/obj/usr/src/amd64.amd64/sys/GENERIC;  MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE= CC="cc -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  AS="as" AR="ar" LD="ld" LLVM_LINK=""  NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  make  KERNEL=kernel install

make[2] warning: /usr/obj/usr/src/amd64.amd64/sys/GENERIC: Permission denied.

ld-elf.so.1: /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin/make: Undefined symbol "regcomp@FBSD_1.6"

*** Error code 1


Stop.

make[1]: stopped in /usr/src

*** Error code 1

The procedure of checkout, buildworld , buildkernel ( generic ) is successfully complete on vm running FreeBSD 13.0.
The shared directories from build machine /usr/src and /usr/obj are readonly.
The installation of kernel and world can be performed without issues when I try it on a FreeBSD 13.0 .
I test it on a FreeBSD 13.0 fresh install. ( Boot mount /usr/src /usr/obj make installkernel and so on.. )
Is there some procedure to take place on FreeBSD12.2 before the upgrade to 13.0?

Thanks in advance.
 
Did you buildkernel and buildworld on the host beforehand? The whole 13.0 thing has to be built on the host. Then you can mount /usr/src/ and /usr/obj on your "client" and run installkernel/installworld there.
 
Yes the host is FreeBSD 13.0 the world and kernel are built there without any issues. The "client" system is FreeBSD12.2 and suppose to perform the standard procedure (make installkernel reboot etcupdate -p make installworld, etcupdate -B and so on. If the "clietnt" is FreeBSD 13.0 the installation of compiled world and kernel can be performed with no issues.
 
The procedure of checkout, buildworld , buildkernel ( generic ) is successfully complete on vm running FreeBSD 13.0.
And that's the problem. The build also builds tools (like make) below /usr/obj, which are then used. This is necessary for "bootstrapping". If these tools are built on 13, they will not run on 12.x.

Sure you can make installworld and make installkernel on a different machine than you did the builds, but it won't work if you built on a newer major release.
 
If the 'old' machine is a 12.2-RELEASE why don't you use freebsd-update(8) instead?

Anyway, I think you can do this if you use a 12.2 host and build the 13.0 world/kernel there. Then mount that on the "client" machine. That should make sure the bootstraps are built for 12.2 and the upgrade should work.
 
This sounds more reasonable I will upgrade a FreeBSD 12.2 to 13.0 and then I will upgrade a "12.2 client" to 13.0 . The reason of using compiled sources is the pf_alq kernel option. Thank you all very mach. I report back the results.
 
The reason of using compiled sources is the pf_alq kernel option.
Doesn't need to be compiled in any more as far as I know. The pf.ko module will have it turned on.

The reason of using compiled sources is the pf_alq kernel option.
You don't need to upgrade the whole system from source to build a custom kernel. Just switch back to the GENERIC kernel temporarily, do the upgrade with freebsd-update(8) and then only rebuild your custom kernel from source. If there are sources in /usr/src/ a freebsd-update(8) will update those too. But you could always just checkout those sources with git or gitup afterwards.
 
The same happened to me a while ago. I only install from source (I have a long list of things I leave out from base, for example), and I have a dedicated machine for building, with all other machines installing it from NFS mounts.

I could solve my problem because I still had a 12.2 base in a dataset I could null-mount to create a jail and rebuild there into a clean /usr/obj, which was then installable on the machines still running 12.

But from that experience, yes, it's a big fat warning for everyone using the NFS approach to install custom builds on a park of machines. With a major release upgrade (when the ABI changes), make sure to have the upgrade installed on all the machines before doing another build :)

Maybe, some day, pkg-base will be a reality, offering a much better way to distribute your custom builds in your own network :)

Until then, SirDice is right of course: If all you want is a customized kernel, there's nothing wrong with just using binary installations/upgrades :) And if you want/need custom builds, make sure to keep the warning above in mind.
 
I've done this a million times with 3-STABLE, 4-STABLE, 5-STABLE, etc. But now that you mentioned it, it was always done with a major version upgrade I had built on a machine I already did the upgrade on (so the bootstrap was built for the upgrade too). Never realized that.
 
So I start all this form scratch in order to have more notes and details about the process. ( This is the testing environment anyway )

The builder vm is a FreeBSD 12.2 fully updated fresh source checkout and nfs sharing in read only mode /usr/src and /usr/obj
the world and kernel will for now not be insalled on this machine.

Builder state
Code:
uname -a && freebsd-version -uk
FreeBSD fb122 12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6 GENERIC  amd64
12.2-RELEASE-p6
12.2-RELEASE-p6

Lets make the kernel without build the world (The BOOK says build the world and then a kernel )
Code:
make -j8 buildkernel
--- buildkernel ---
make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: libclang will be built for bootstrapping a cross-linker.
--- buildkernel ---
--------------------------------------------------------------
>>> Kernel build for GENERIC started on Tue May 18 19:02:28 EEST 2021
--------------------------------------------------------------
===> GENERIC
mkdir -p /usr/obj/usr/src/amd64.amd64/sys
--------------------------------------------------------------
>>> stage 1: configuring the kernel
--------------------------------------------------------------
cd /usr/src/sys/amd64/conf;  PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  config  -d /usr/obj/usr/src/amd64.amd64/sys/GENERIC  -I '/usr/src/sys/amd64/conf' -I '/usr/src/sys/amd64/conf'  '/usr/src/sys/amd64/conf/GENERIC'
ERROR: version of config(8) does not match kernel!
config version = 600016, version required = 600018

Make sure that /usr/src/usr.sbin/config is in sync
with your /usr/src/sys and install a new config binary
before trying this again.

If running the new config fails check your config
file against the GENERIC or LINT config files for
changes in config syntax, or option/device naming
conventions

As expected so buildworld then kernel..
Code:
--- buildworld_epilogue ---
--------------------------------------------------------------
>>> World build completed on Tue May 18 20:21:19 EEST 2021
>>> World built in 4668 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
lets install GENERIC kernel whitout build one ...
Code:
make installkernel
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
--------------------------------------------------------------
>>> Installing kernel GENERIC on Tue May 18 22:03:30 EEST 2021
--------------------------------------------------------------
cd /usr/obj/usr/src/amd64.amd64/sys/GENERIC;  MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE= CC="cc -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  AS="as" AR="ar" LD="ld" LLVM_LINK=""  NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  make  KERNEL=kernel install
cd: /usr/obj/usr/src/amd64.amd64/sys/GENERIC: No such file or directory
*** Error code 2

Stop.
make[1]: stopped in /usr/src
*** Error code 1

So the kernel must be build! (As the BOOK says! )
Code:
--------------------------------------------------------------
>>> Kernel build for GENERIC completed on Tue May 18 22:11:21 EEST 2021
--------------------------------------------------------------
>>> Kernel(s)  GENERIC built in 413 seconds, ncpu: 8, make -j8
--------------------------------------------------------------

Nicely done! now lets install it on client machine!
Code:
make kernelinstall

* * *
===> zlib (install)
install -T release -o root -g wheel -m 555   zlib.ko /boot/kernel/
install -T dbg -o root -g wheel -m 555   zlib.ko.debug /usr/lib/debug/boot/kernel/
kldxref /boot/kernel
kldxref: /boot/kernel/kernel: too many segments
--------------------------------------------------------------
>>> Installing kernel GENERIC completed on Tue May 18 22:12:36 EEST 2021
--------------------------------------------------------------
reboot
uname -a 
FreeBSD fbtest12 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Tue May 18 22:07:11 EEST 2021     root@fb122:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
root@fbtest12:~ # freebsd-version -uk
13.0-RELEASE
12.2-RELEASE

Lets install the fresh compiled world into client machine!
Code:
etcupdate -p
cd /usr/src
make installworld
etcupdate -B

World installed successfully no issues! lets see before reboot...
Code:
freebsd-version -uk
13.0-RELEASE
13.0-RELEASE
Lets clean old libs etc so mount and
Code:
cd /usr/src
make check-old
yes | make delete-old
yes | make delete-old-libs
reboot
No issues! moving on!! What about ZFS?
Code:
root@fbtest12:~ # zpool status  
  pool: zroot                                
 state: ONLINE                                                                                                                                                                                                     
status: Some supported features are not enabled on the pool. The pool can                                
        still be used, but some features are unavailable.                                                
action: Enable all features using 'zpool upgrade'. Once this is done,                                    
        the pool may no longer be accessible by software that does not support                           
        the features. See zpool-features(5) for details.
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          vtbd0p3   ONLINE       0     0     0

errors: No known data errors
root@fbtest12:~ # zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  userobj_accounting
  encryption
  project_quota
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid

root@fbtest12:~ # zpool status
  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          vtbd0p3   ONLINE       0     0     0

errors: No known data errors
root@fbtest12:~ # shutdown -r now
Everything works perfect! `no warrings no signs` at dmesg The clinet machine is upgraded to FreeBSD13.0
Ok so the PF ALTQ module is on kernel? No need to make a custom one? Lets see..
Code:
ext_if="vtnet0"
main_tcp_ports= " { 43 53 80 443 } "
main_udp_ports= " { 53 123 }"

TcpState ="flags S/SA keep state"
UdpState ="keep state"
set state-policy if-bound 

set skip on { lo0  }

scrub on re1 all fragment reassemble 

altq on $ext_if bandwidth 4498Kb hfsc queue { ack, main, bulk }
  queue ack     bandwidth 1100Kb qlimit 500 hfsc (realtime 1000Kb upperlimit 99%  )
  queue main    bandwidth 2000Kb qlimit 500 hfsc ( default realtime 1900Kb upperlimit 99%)
  queue bulk    bandwidth 1000Kb qlimit 500 hfsc (upperlimit 99%)

nat on $ext_if inet from !($ext_if) to any -> ($ext_if)
no rdr
antispoof log for { $ext_if }

block in log all
block return out quick on $ext_if proto ipv6

pass out quick on $ext_if inet proto tcp from any to any port $main_tcp_ports $TcpState queue (main, ack)
pass out quick on $ext_if inet proto tcp from any to any $TcpState queue (bulk, ack)
pass out quick on $ext_if inet proto icmp from any to any $UdpState #queue ( bulk )
pass out quick on $ext_if inet proto udp from any to any port $main_udp_ports $UdpState queue (main)
pass out quick on $ext_if inet proto udp from any to any $UdpState queue (bulk)
pass in quick log on $ext_if inet proto icmp from any to any $UdpState
Check syntax and load rules
Code:
pfctl -nf /etc/pf.conf
No issues, now load the rules
Code:
pfctl -f /etc/pf.conf
altq not defined on vtnet0
altq not defined on vtnet0
pf.conf:14: errors in queue definition
altq not defined on vtnet0
pf.conf:15: errors in queue definition
altq not defined on vtnet0
pf.conf:16: errors in queue definition
pfctl: Syntax error in config file: pf rules not loaded
So we need a custom kernel for alt hfsc to work.
Here is the PF_ALTQ file that I use to buld the custom kernel
Code:
include GENERIC
ident PF_ALTQ
##PF_ALTQ Settings
device pf
device pflog
device pfsync
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build
The kernel was sucsesfuly buld on build machine but I was not able to installed it on client!
Code:
>>> Kernel build for PF_ALTQ completed on Tue May 18 23:04:36 EEST 2021
--------------------------------------------------------------
>>> Kernel(s)  PF_ALTQ built in 409 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
So on client machine
Code:
make installkernel KERNCONF=PF_ALTQ
make warning: /usr/obj/usr/src: Permission denied.
make[1] warning: /usr/src/: Permission denied.
make[2] warning: /usr/obj/usr/src/amd64.amd64: Permission denied.
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
ERROR: No kernel "PF_ALTQ" to install.
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src
The same results I had when I make the custom kernel install on fresh FreeBSD13.0 vm.
So now the issue is that I cannot install the custom kernel on client systems.I dont make any changes on bulder any idea for what next?
 
Lets make the kernel without build the world (The BOOK says build the world and then a kernel )
You can actually do this but you would need to make kernel-toolchain first.
Code:
     kernel-toolchain      Rebuild the tools needed for kernel compilation.
                           Use this if you did not do a buildworld first.

Code:
yes | make delete-old 
yes | make delete-old-libs
make -DBATCH_DELETE_OLD_FILES delete-old
make -DBATCH_DELETE_OLD_FILES delete-old-libs

Code:
     delete-old           Delete obsolete base system files and directories
                          interactively.  When -DBATCH_DELETE_OLD_FILES is
                          specified at the command line, the delete operation
                          will be non-interactive.  The variables DESTDIR,
                          TARGET_ARCH and TARGET should be set as with “make
                          installworld”.

See build(7).
 
Thank you very much for your time and answers! I had not spot the build manual page before. All the info about build operations was there.
I manage to install the custom kernel without any issues. The kernel configuration file was symbolic link on nfs server and was not available on client machine!
On the production I had the configuration file ( /root/PF_ALTQ ) and was never had this issue. Symbolic are not valid in nfs clients.. leaned that on hard way.
 
Back
Top