So I start all this form scratch in order to have more notes and details about the process. ( This is the testing environment anyway )
The builder vm is a FreeBSD 12.2 fully updated fresh source checkout and nfs sharing in read only mode /usr/src and /usr/obj
the world and kernel will for now not be insalled on this machine.
Builder state
Code:
uname -a && freebsd-version -uk
FreeBSD fb122 12.2-RELEASE-p6 FreeBSD 12.2-RELEASE-p6 GENERIC amd64
12.2-RELEASE-p6
12.2-RELEASE-p6
Lets make the kernel without build the world (The BOOK says build the world and then a kernel )
Code:
make -j8 buildkernel
--- buildkernel ---
make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: libclang will be built for bootstrapping a cross-linker.
--- buildkernel ---
--------------------------------------------------------------
>>> Kernel build for GENERIC started on Tue May 18 19:02:28 EEST 2021
--------------------------------------------------------------
===> GENERIC
mkdir -p /usr/obj/usr/src/amd64.amd64/sys
--------------------------------------------------------------
>>> stage 1: configuring the kernel
--------------------------------------------------------------
cd /usr/src/sys/amd64/conf; PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin config -d /usr/obj/usr/src/amd64.amd64/sys/GENERIC -I '/usr/src/sys/amd64/conf' -I '/usr/src/sys/amd64/conf' '/usr/src/sys/amd64/conf/GENERIC'
ERROR: version of config(8) does not match kernel!
config version = 600016, version required = 600018
Make sure that /usr/src/usr.sbin/config is in sync
with your /usr/src/sys and install a new config binary
before trying this again.
If running the new config fails check your config
file against the GENERIC or LINT config files for
changes in config syntax, or option/device naming
conventions
As expected so buildworld then kernel..
Code:
--- buildworld_epilogue ---
--------------------------------------------------------------
>>> World build completed on Tue May 18 20:21:19 EEST 2021
>>> World built in 4668 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
lets install GENERIC kernel whitout build one ...
Code:
make installkernel
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
--------------------------------------------------------------
>>> Installing kernel GENERIC on Tue May 18 22:03:30 EEST 2021
--------------------------------------------------------------
cd /usr/obj/usr/src/amd64.amd64/sys/GENERIC; MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE= CC="cc -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" AS="as" AR="ar" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel install
cd: /usr/obj/usr/src/amd64.amd64/sys/GENERIC: No such file or directory
*** Error code 2
Stop.
make[1]: stopped in /usr/src
*** Error code 1
So the kernel must be build! (As the BOOK says! )
Code:
--------------------------------------------------------------
>>> Kernel build for GENERIC completed on Tue May 18 22:11:21 EEST 2021
--------------------------------------------------------------
>>> Kernel(s) GENERIC built in 413 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
Nicely done! now lets install it on client machine!
Code:
make kernelinstall
* * *
===> zlib (install)
install -T release -o root -g wheel -m 555 zlib.ko /boot/kernel/
install -T dbg -o root -g wheel -m 555 zlib.ko.debug /usr/lib/debug/boot/kernel/
kldxref /boot/kernel
kldxref: /boot/kernel/kernel: too many segments
--------------------------------------------------------------
>>> Installing kernel GENERIC completed on Tue May 18 22:12:36 EEST 2021
--------------------------------------------------------------
reboot
uname -a
FreeBSD fbtest12 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Tue May 18 22:07:11 EEST 2021 root@fb122:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
root@fbtest12:~ # freebsd-version -uk
13.0-RELEASE
12.2-RELEASE
Lets install the fresh compiled world into client machine!
Code:
etcupdate -p
cd /usr/src
make installworld
etcupdate -B
World installed successfully no issues! lets see before reboot...
Code:
freebsd-version -uk
13.0-RELEASE
13.0-RELEASE
Lets clean old libs etc so mount and
Code:
cd /usr/src
make check-old
yes | make delete-old
yes | make delete-old-libs
reboot
No issues! moving on!! What about ZFS?
Code:
root@fbtest12:~ # zpool status
pool: zroot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
vtbd0p3 ONLINE 0 0 0
errors: No known data errors
root@fbtest12:~ # zpool upgrade zroot
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
userobj_accounting
encryption
project_quota
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid
root@fbtest12:~ # zpool status
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
vtbd0p3 ONLINE 0 0 0
errors: No known data errors
root@fbtest12:~ # shutdown -r now
Everything works perfect! `no warrings no signs` at dmesg The clinet machine is upgraded to FreeBSD13.0
Ok so the PF ALTQ module is on kernel? No need to make a custom one? Lets see..
Code:
ext_if="vtnet0"
main_tcp_ports= " { 43 53 80 443 } "
main_udp_ports= " { 53 123 }"
TcpState ="flags S/SA keep state"
UdpState ="keep state"
set state-policy if-bound
set skip on { lo0 }
scrub on re1 all fragment reassemble
altq on $ext_if bandwidth 4498Kb hfsc queue { ack, main, bulk }
queue ack bandwidth 1100Kb qlimit 500 hfsc (realtime 1000Kb upperlimit 99% )
queue main bandwidth 2000Kb qlimit 500 hfsc ( default realtime 1900Kb upperlimit 99%)
queue bulk bandwidth 1000Kb qlimit 500 hfsc (upperlimit 99%)
nat on $ext_if inet from !($ext_if) to any -> ($ext_if)
no rdr
antispoof log for { $ext_if }
block in log all
block return out quick on $ext_if proto ipv6
pass out quick on $ext_if inet proto tcp from any to any port $main_tcp_ports $TcpState queue (main, ack)
pass out quick on $ext_if inet proto tcp from any to any $TcpState queue (bulk, ack)
pass out quick on $ext_if inet proto icmp from any to any $UdpState #queue ( bulk )
pass out quick on $ext_if inet proto udp from any to any port $main_udp_ports $UdpState queue (main)
pass out quick on $ext_if inet proto udp from any to any $UdpState queue (bulk)
pass in quick log on $ext_if inet proto icmp from any to any $UdpState
Check syntax and load rules
No issues, now load the rules
Code:
pfctl -f /etc/pf.conf
altq not defined on vtnet0
altq not defined on vtnet0
pf.conf:14: errors in queue definition
altq not defined on vtnet0
pf.conf:15: errors in queue definition
altq not defined on vtnet0
pf.conf:16: errors in queue definition
pfctl: Syntax error in config file: pf rules not loaded
So we need a custom kernel for alt hfsc to work.
Here is the PF_ALTQ file that I use to buld the custom kernel
Code:
include GENERIC
ident PF_ALTQ
##PF_ALTQ Settings
device pf
device pflog
device pfsync
options ALTQ
options ALTQ_CBQ # Class Bases Queuing (CBQ)
options ALTQ_RED # Random Early Detection (RED)
options ALTQ_RIO # RED In/Out
options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC)
options ALTQ_PRIQ # Priority Queuing (PRIQ)
options ALTQ_NOPCC # Required for SMP build
The kernel was sucsesfuly buld on build machine but I was not able to installed it on client!
Code:
>>> Kernel build for PF_ALTQ completed on Tue May 18 23:04:36 EEST 2021
--------------------------------------------------------------
>>> Kernel(s) PF_ALTQ built in 409 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
So on client machine
Code:
make installkernel KERNCONF=PF_ALTQ
make warning: /usr/obj/usr/src: Permission denied.
make[1] warning: /usr/src/: Permission denied.
make[2] warning: /usr/obj/usr/src/amd64.amd64: Permission denied.
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
ERROR: No kernel "PF_ALTQ" to install.
*** Error code 1
Stop.
make[1]: stopped in /usr/src
*** Error code 1
Stop.
make: stopped in /usr/src
The same results I had when I make the custom kernel install on fresh FreeBSD13.0 vm.
So now the issue is that I cannot install the custom kernel on client systems.I dont make any changes on bulder any idea for what next?