Installing kernel fails from crossbuilt i386.i386

G

getopt

␢
Having done buildworld and buildkernel on an NFS server (amd64), the installkernel process fails like this on an i386 NFS client-mounted src/ and obj/ in /FreeBSD with exact same softlinks in /usr on both machines (development()):

Code: 
# setenv MAKEOBJDIRPREFIX=/FreeBSD/obj/i386.i386
# cd /FreeBSD/src
# make installkernel 
--------------------------------------------------------------
>>> Installing kernel GENERIC
--------------------------------------------------------------
cd /FreeBSD/obj/i386.i386/FreeBSD/src/sys/GENERIC;  MAKEOBJDIRPREFIX=/FreeBSD/obj/i386.i386  MACHINE_ARCH=i386  MACHINE=i386  CPUTYPE= GROFF_BIN_PATH=/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/bin  GROFF_FONT_PATH=/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/share/groff_font  GROFF_TMAC_PATH=/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/share/tmac PATH=/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/sbin:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/bin:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/usr/games:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/legacy/bin:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/usr/sbin:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/usr/bin:/FreeBSD/obj/i386.i386/FreeBSD/src/tmp/usr/games:/sbin:/bin:/usr/sbin:/usr/bin  make  KERNEL=kernel install
*** Error code 1

Stop.
make[1]: stopped in /FreeBSD/src
*** Error code 1

Stop.
make: stopped in /FreeBSD/src

I’m stuck on this, please help.
 
I've tackled with the same problem and there are two solutions. First is to use -D CROSS_BUILD_TESTING on the make(1) command line on the client, that will force it to use the i386.i386 subdirectory even if in its view you're not cross-compiling. The second solution is a bit more elaborate, it is to use a jail(8) for the i386 build. I have opted to use the second solution and here is a rough sketch of what I have:


The jail.conf(5) entry for the jail:

Code: 
# Typical static defaults:
# Use the rc scripts to start and stop jails.  Mount jail's /dev.
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

# Dynamic wildcard parameter:
# Base the path off the jail name.
path = "/data/jails/$name";

build-stable-10-i386 {
    host.hostname = "$name.rdnzl.info";
    exec.start = "";
    exec.stop = "";
    allow.chflags = "true";
    devfs_ruleset = 0;
    persist;
}

Note that the settings are very permissive and this jail wouldn't be very secure for running services but it's only for building in my case.

I have the system sources mounted in the jail using nullfs(5), the entry in my /etc/fstab:

Code: 
/data/src/stable/10 /data/jails/build-stable-10-i386/usr/src nullfs defaults,ro,late 0 0

Note that my /usr/src is empty usually and the system sources are in non-standard location /data/src/stable/10, I'm using the same exact set up for building the amd64 world and kernel to separate all building activities to jails.

The sources and the /usr/obj directory of the jail are NFS exported:

Code: 
/data/src/stable/10 -ro -maproot=root -network 10.71.14/24
/data/src/stable/10 -ro -maproot=root -network 2001:14b8:XXXX:YYYY::/64
/data/jails/build-stable-10-i386/usr/obj -ro -maproot=root -network 10.71.14/24
/data/jails/build-stable-10-i386/usr/obj -ro -maproot=root -network 2001:14b8:XXXX:YYYY::/64

Those are then NFS mounted on the client as you would expect. The exports are read-only to guarantee that the client can not do changes on the mounted shares.

Building the world and kernel for i386 is then as simple as:

Code: 
jail -c build-stable-10-i386
jexec build-stable-10-i386 sh
cd /usr/src
make -j4 buildworld buildkernel


The beauty of this solution is that I can use the same jail as a jail for ports-mgmt/poudriere-devel to build packages for my i386 host, no need for many jails for different purposes that would be still identical in contents.

(I should probably turn these instructions in to a HOWTO...)
 
Oh snap..., yes the CROSS_BUILD_TESTING method doesn't really work because the build tools built by the first stage of make buildworld are still native amd64 binaries. I now remember trying it myself but failing exactly as you did.

The crossbuilding works without a jail as long as your target is just building world and kernel for release(7) sets for i386. Crossbuilding is also a way to bootstrap an i386 jail for the method I described above, however you will get going much faster using a snapshot from ftp://ftp.freebsd.org/pub/FreeBSD/snapshots.
 
getopt said:
Hmm, does that mean, the method lengthily described in man 7 development() is not supposed to be extended for cross building?
Click to expand...

You can't export the /usr/obj directory to a client with different architecture because of the limitations mentioned above. Let's say you're crossbuilding for powerpc. The crossbuild process has to create the cross-compilation tools using the native executable format (amd64) of the host system, otherwise they can't be run. Unfortunately the same tools are also assumed to be in place at the time of installation and be of the native format for the host architecture. This means that if you mount the amd64 /usr/obj on a powerpc host the installworld/installkernel targets will assume that the build tools are in native powerpc format which they aren't because the cross-compilation required them to be in amd64 format.
 
getopt said:
The need for custom kernels excludes the build release method (true?).
Click to expand...
Nope. A release build will build all kernels that are set with KERNCONF, the first kernel on the list will be used as default. The other kernels will be built and saved as kernel-<CONFNAME>.tgz.

I have KERNCONF set in /etc/make.conf:
Code: 
KERNCONF=MOLLY CORTO MAELCUM GENERIC
This will build all kernels and uses MOLLY as its default. You can override this by setting KERNCONF on the command line with make release.

Code: 
dice@molly:~ % ll /storage/release/10-stable/ftp/
total 356678
-rw-r--r--  1 root  wheel        830 Jun 28 18:28 MANIFEST
-rw-r--r--  1 root  wheel   64836236 Jun 28 18:28 base.txz
-rw-r--r--  1 root  wheel    1429480 Jun 28 18:28 doc.txz
-rw-r--r--  1 root  wheel     885988 Jun 28 18:28 games.txz
-rw-r--r--  1 root  wheel   15447072 Jun 28 18:28 kernel.CORTO.txz
-rw-r--r--  1 root  wheel   88062708 Jun 28 18:28 kernel.GENERIC.txz
-rw-r--r--  1 root  wheel   14812080 Jan 30 20:50 kernel.MAELCUM.txz
-rw-r--r--  1 root  wheel   15373008 Jun 28 18:28 kernel.txz
-rw-r--r--  1 root  wheel   16307928 Jun 28 18:28 lib32.txz
-rw-r--r--  1 root  wheel   34642744 Mar 20 17:36 ports.txz
-rw-r--r--  1 root  wheel  112911804 Mar 20 17:36 src.txz
 
getopt said:
Thanks @kpa for that explanation.

The general problem is, there are a fast amd64 machine and some slow i386s which need custom kernels. The need for custom kernels excludes the build release method (true?). As frequency of relengs can be a few days only there is a need for speeding the builds for i386 machines, as they run a half day minimum, up to a whole day for building world and kernel. I’m quite sure that I’m not alone with this problem. So your method with the jail is of great interest, as I’m already running some jails on the amd64 machine, using ZFS and ezjail. If I got you right, you are running an i386-jail on an amd64 machine? In that jail which has ARCH=i386 the build process is done? This builds by i386 can then be nfs_exported to i386 clients.
Click to expand...

Yes, the building of world and kernel for i386 is done in a jail that is completely i386. That guarantees that everything in the /usr/obj directory of the jail is for i386. The /usr/obj directory of the jail is then NFS exported to the i386 host(s) as I showed above.
 
Last edited by a moderator:
The jails registered/created by ports-mgmt/poudriere are not running by default but are started and stopped automatically when needed. There will be quite a few "clones" of the jails as well when the package builder is running, you can see them with jls(8). Those jails that you have now were created by ports-mgmt/poudriere and should not be used for other purposes.

I'm doing my builds the "opposite" way by first creating the jails that I use for build(7) the way I want and setting them up in /etc/jail.conf. I'm then telling ports-mgmt/poudriere-devel (the non-devel version can not be used unfortunately) that the jails it should use for building the packages are located at the directories used by the build(7) jails. This is done with the -m null -M /path/to/jail options of poudriere. This method works but has turned out to be quite tricky to get right because poudriere does some customizations to the jails at creation time and creates a @clean snapshot of the jail ZFS filesystem that has to be updated every time you do an OS update on the jail.
 
You must log in or register to reply here.
Back
Top