What is the correct procedure to use bectl boot-environments with make installworld

Alain De Vos · Apr 5, 2022

What is the correct procedure to use bectl boot-environments together with make installworld and make installkernel ?
And what is the procedure to go back in case of early boot crashes ?

Eric A. Borisch · Apr 5, 2022

I use a jail in a mounted (new; not at /) BE for installworld/installkernel. (beadm mount + a minimal jail config / fstab) … build*s are done in the active environment first.

That said, you *can* just always keep running in a latest BE, and just create a BE of the about-to-be-replaced version before installworld/installkernel. Dealer’s choice.

Early (pre-userland) failures generally require using the loader to select a working BE; there might be some clever way with nextboot(8), but I’ve never found it.

Andriy · Apr 5, 2022

DESTDIR is what you can use to install to an alternative root.
Or chroot / jail (if needed, with nullfs mounts of /usr/src and /usr/obj) as suggested earlier.

Cath O'Deray · Apr 10, 2022

Andriy said:
DESTDIR is what you can use to install to an alternative root. …

After mounting the boot environment, whilst installing the kernel, Unable to locate the kernel source tree:

Code:

…
Successfully mounted n254628-a3214fbe7fff8b4914f12e187dcc972e52db548d-a at /tmp/up
root@mowa219-gjp4-8570p-freebsd:/usr/src # cd /usr/src && time make installkernel KERNCONF=GENERIC-NODEBUG DESTDIR=/tmp/up >& installkernel.log && history -S
50.476u 15.808s 3:22.62 32.7%   3802+290k 26625+71158io 6674pf+0w
root@mowa219-gjp4-8570p-freebsd:/usr/src # grep SYSDIR installkernel.log
cd /usr/src/sys/modules; MAKEOBJDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG/modules KMODDIR=/boot/kernel MACHINE_CPUARCH=amd64 MACHINE=amd64 MACHINE_ARCH=amd64 MODULES_EXTRA="" WITHOUT_MODULES="" ARCH_FLAGS="" DEBUG_FLAGS="-g" __MPATH="" DESTDIR="/tmp/up" KERNBUILDDIR="/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG" SYSDIR="/usr/src/sys" MODULE_TIED=yes WITH_CTF="1" COMPAT_FREEBSD32_ENABLED="yes" make  install
cd ${PORTSDIR:-/usr/ports}/emulators/virtualbox-ose-kmod; env  -u CC  -u CXX  -u CPP  -u MAKESYSPATH  -u MK_AUTO_OBJ  -u MAKEOBJDIR  MAKEFLAGS="DESTDIR=/tmp/up KERNCONF=GENERIC-NODEBUG KERNEL=kernel MK_META_MODE=no TARGET=amd64 TARGET_ARCH=amd64"  SYSDIR=/usr/src/sys  PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin  SRC_BASE=/usr/src  OSVERSION=1400056  WRKDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG make -B deinstall reinstall
make[4]: "/usr/share/mk/bsd.sysdir.mk" line 15: Unable to locate the kernel source tree. Set SYSDIR to override.
root@mowa219-gjp4-8570p-freebsd:/usr/src #

– in context:

Code:

…
kBuild: Installing /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/out/freebsd.amd64/release/bin/src/vboxnetadp/Makefile
kBuild: Installing /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/out/freebsd.amd64/release/bin/src/vboxdrv/Makefile
--- all ---
=== Building 'vboxdrv' module ===
make[4]: "/usr/share/mk/bsd.sysdir.mk" line 15: Unable to locate the kernel source tree. Set SYSDIR to override.

make[4]: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/out/freebsd.amd64/release/bin/src/vboxdrv
*** [all] Error code 1

make[3]: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/out/freebsd.amd64/release/bin/src
1 error

make[3]: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/out/freebsd.amd64/release/bin/src
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[2]: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[1]: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make: stopped in /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod
===>  Chrooted make in /tmp/up failed
===>  Cleaning up...
*** Error code 1

Stop.
make[3]: stopped in /usr/ports/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[2]: stopped in /usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

With SYSDIR=/usr/src/sys, a different issue:

Code:

…
===>  Building for virtualbox-ose-kmod-6.1.32_1
cd /tmp/mountpoint.lEZzQL/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/src/VBox/HostDrivers && /bin/sh -c  '. /tmp/mountpoint.lEZzQL/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/env.sh && VBOX_LIBPATH_X11=/usr/local VBOX_FREEBSD_SRC=/usr/src/sys /usr/local/bin/kmk HostDrivers-scripts vboxdrv-src VBoxNetFlt-src VBoxNetAdp-src'
Makefile.kmk:28: /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/kBuild/subheader.kmk: No such file or directory
Makefile.kmk:31: /Support/Makefile.kmk: No such file or directory
kmk: *** No rule to make target `/Support/Makefile.kmk'.  Stop.
*** Error code 2

Stop.
make[2]: stopped in /tmp/mountpoint.lEZzQL/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[1]: stopped in /tmp/mountpoint.lEZzQL/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make: stopped in /tmp/mountpoint.lEZzQL/emulators/virtualbox-ose-kmod
===>  Chrooted make in /tmp/up failed
===>  Cleaning up...
*** Error code 1

Stop.
make[3]: stopped in /usr/ports/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[2]: stopped in /usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

/etc/src.conf

KERNCONF=GENERIC-NODEBUG
PORTS_MODULES= emulators/virtualbox-ose-kmod graphics/drm-devel-kmod graphics/gpu-firmware-kmod
WITH_CCACHE_BUILD=yes
WITH_MALLOC_PRODUCTION=yes

Cath O'Deray · Apr 10, 2022

beinstall.sh(8)

The same issue. What am I missing?

Code:

root@mowa219-gjp4-8570p-freebsd:~ # /usr/src/tools/build/beinstall.sh KERNCONF=GENERIC-NODEBUG
error: Must be in FreeBSD source tree
root@mowa219-gjp4-8570p-freebsd:~ # cd /usr/src
root@mowa219-gjp4-8570p-freebsd:/usr/src # /usr/src/tools/build/beinstall.sh KERNCONF=GENERIC-NODEBUG
Successfully mounted 14.0-CURRENT-20220409.165943 at /tmp/beinstall.wdL42J/mnt
Mounted 14.0-CURRENT-20220409.165943 to /tmp/beinstall.wdL42J/mnt, performing install/update ...
--------------------------------------------------------------
>>> Install check kernel
--------------------------------------------------------------
--------------------------------------------------------------
>>> Installing kernel GENERIC-NODEBUG on Sun Apr 10 16:53:20 BST 2022
--------------------------------------------------------------
…
===>  Building for virtualbox-ose-kmod-6.1.32_1
cd /tmp/mountpoint.kVhX2N/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/src/VBox/HostDrivers && /bin/sh -c  '. /tmp/mountpoint.kVhX2N/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/env.sh && VBOX_LIBPATH_X11=/usr/local VBOX_FREEBSD_SRC=/usr/src/sys /usr/local/bin/kmk HostDrivers-scripts vboxdrv-src VBoxNetFlt-src VBoxNetAdp-src'
Makefile.kmk:28: /tmp/mountpoint.EPGdT8/emulators/virtualbox-ose-kmod/work/VirtualBox-6.1.32/kBuild/subheader.kmk: No such file or directory
Makefile.kmk:31: /Support/Makefile.kmk: No such file or directory
kmk: *** No rule to make target `/Support/Makefile.kmk'.  Stop.
*** Error code 2

Stop.
make[2]: stopped in /tmp/mountpoint.kVhX2N/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[1]: stopped in /tmp/mountpoint.kVhX2N/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make: stopped in /tmp/mountpoint.kVhX2N/emulators/virtualbox-ose-kmod
===>  Chrooted make in /tmp/beinstall.wdL42J/mnt failed
===>  Cleaning up...
*** Error code 1

Stop.
make[3]: stopped in /usr/ports/emulators/virtualbox-ose-kmod
*** Error code 1

Stop.
make[2]: stopped in /usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src
Cleaning up ...
umount -f /tmp/beinstall.wdL42J/mnt
error: Installkernel failed!
root@mowa219-gjp4-8570p-freebsd:/usr/src #

Cath O'Deray · Apr 10, 2022

grahamperrin said:
The same issue. What am I missing?

For now (without an answer) I commented out the PORTS_MODULES line in my /etc/src.conf.

The next failure was more predictable. Wishing to work around with NO_PKG_UPGRADE=yes:

Code:

root@mowa219-gjp4-8570p-freebsd:/usr/src # /usr/src/tools/build/beinstall.sh KERNCONF=GENERIC-NODEBUG
…
Skipping untrusted certificate /usr/share/certs/trusted/thawte_Primary_Root_CA_-_G3.pem (/etc/ssl/untrusted/ba89ed3b.0)
Scanning /usr/local/share/certs for certificates...
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
pkg: file:///usr/local/poudriere/data/packages/main-default/meta.txz: No such file or directory
repository poudriere has no meta file, using default settings
pkg: file:///usr/local/poudriere/data/packages/main-default/packagesite.pkg: No such file or directory
pkg: file:///usr/local/poudriere/data/packages/main-default/packagesite.txz: No such file or directory
Unable to update repository poudriere
Error updating repositories!
Cleaning up ...
Rsyncing back newly saved packages...
sending incremental file list

sent 51 bytes  received 12 bytes  126.00 bytes/sec
total size is 0  speedup is 0.00
umount -f /tmp/beinstall.qQWfQ5/mnt/usr/src /tmp/beinstall.qQWfQ5/mnt/usr/obj/usr/src/amd64.amd64 /tmp/beinstall.qQWfQ5/mnt/dev /tmp/beinstall.qQWfQ5/mnt
error: Unable to update pkg
root@mowa219-gjp4-8570p-freebsd:/usr/src # /usr/src/tools/build/beinstall.sh KERNCONF=GENERIC-NODEBUG NO_PKG_UPGRADE=yes

During the run:

Code:

% ioztat -z -i 10 
                                           operations    throughput 
dataset                                    read  write   read  write
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943     45      7  86.4K   391K
august/ROOT/n254268-50e244964e9-d           127      8  1.22M   427K
august/poudriere/data/cache                   0      0      1      0
august/poudriere/data/logs                  106      4  28.7K  28.2K
august/poudriere/data/packages                4      0  64.3K  1.46K
august/poudriere/jails/main                 304      7  1.70M   328K
august/poudriere/ports/default                1      0  19.1K  8.10K
august/usr/home                             441    118  7.06M   371K
august/usr/ports                              0      0  37.5K  37.3K
august/usr/src                              158      0   619K      0
august/var/log                                0      0    186     61
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943    694     94   931K  2.41M
august/ROOT/n254268-50e244964e9-d            98      0   635K      0
august/usr/home                              30    339  83.3K   136K
august/usr/src                            1.24K      0  5.29M      0
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943  1.17K    198  1.61M  4.70M
august/ROOT/n254268-50e244964e9-d            62      0   195K      0
august/usr/home                              31    612  85.0K   418K
august/usr/src                            1.63K      0  6.74M      0
august/var/log                                0      0      0     13
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943    495     45   743K   475K
august/ROOT/n254268-50e244964e9-d            72      0   337K      0
august/usr/home                              19    204  51.1K   101K
august/usr/src                            1.01K      0  4.11M      0
august/var/log                                0      0      0     13
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943    339      9   781K   388K
august/ROOT/n254268-50e244964e9-d             5      0    854      0
august/usr/home                               0      0  1.62K      0
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943    353     60   835K  3.57M
august/ROOT/n254268-50e244964e9-d            48      0   610K      0
august/usr/home                              27    127  73.4K  47.9K
august/usr/src                              676      0  3.01M      0
----------------------------------------  -----  -----  -----  -----
august/ROOT/14.0-CURRENT-20220409.165943    436     42  1.11M  2.43M
august/ROOT/n254268-50e244964e9-d            71      0  1.09M      0
august/usr/home                              29    198  80.4K  77.0K
august/usr/src                              831      0  3.73M      0
august/var/log                                0      0      0     13
^C
%

Unfortunately, another package upgrade failure occurred; it seemed that NO_PKG_UPGRADE=yes did not skip.

mer · Apr 10, 2022

Different folks have different opinions, but:
bectl create newBE && reboot && make buildworld && make buildkernel && make installkernel && make installworld

That creates a new BE and installs into it.

Other options, which creates a newBE, mounts it, chroots into the BE and upgrades it. Has the advantage of a completely updated BE before a reboot. NOTE: there are some issues with tracking of what happened (/var/log stuff).

Upgrade FreeBSD with ZFS Boot Environments

I am known as a strong ZFS Boot Environment supporter … and not without a reason. I have stated the reasons ‘why’ many times but most (or all) of them are condensed here – &…

vermaden.wordpress.com

I've done both. The important thing is "make sure you are in a new BE before you upgrade. That means rolling back is simply a reboot and selecting a different BE from loader menu"

Cath O'Deray · Apr 10, 2022

FreeBSD bug 263208 – beinstall.sh NO_PKG_UPGRADE="YES" seems to not skip. Updating FreeBSD repository catalogue...

That bug aside, is there any good reason to not mention beinstall.sh in the FreeBSD Handbook?

vermaden were you aware of this script?

freebsd-src/tools/build/beinstall.sh at main · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src

github.com

2017:

dch said:
… if you build from source, there is this amazing gem … which Does The Right Thing - creates a boot environment, installs into it, upgrades the packages in the new BE, and finally merges in any changes to /etc for you. …

2018:

rigoletto@ said:
You may want to take a look on beinstall.sh. Currently it is just available on -CURRENT but should work fine on 11.XR …. This basically wrap sysutils/beadm and automatize the process safely. …

vermaden · Apr 10, 2022

grahamperrin said:
vermaden were you aware of this script?

Yes, but I do not see it useful for *MY* purposes.

Its not that its not useful - it is useful.

For example - to cite the beinstall.sh(8) man page - requires a fully built world and kernel - I am not interested in building world or kernel as they (base.txz and kernel.txz) are available online here for example for 13.1-RC2 version:

- http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RC2/

So I use these files (base.txz and kernel.txz) this (my own) way:

- https://vermaden.wordpress.com/2021/10/19/other-freebsd-version-in-zfs-boot-environment/

Regards.

Cath O'Deray · Apr 14, 2022

building FreeBSD from source into a BE - HackMD

… This last bebuild.sh script is a little wrapper around Will Andrews’ super-duper beinstall.sh script …

Credit: dch <https://people.freebsd.org/~dch/>

Cath O'Deray · Apr 23, 2022

grahamperrin said:
… For now (without an answer) I commented out the PORTS_MODULES line in my /etc/src.conf. …

A comment in Discord:

ports has an incompatible idea of what DESTDIR means, almost certainly this is conflicting

What is the correct procedure to use bectl boot-environments with make installworld

beinstall.sh(8)​

beinstall.sh(8)