Solved how to proper buildworld in 2024?

wolffnx

wolffnx

Hi , I never buildworld because takes me too long, even with the -j switch in a intel corei7 with 8 cores
but I made it..and want to play with it
this is the correct process?

Code: 
 rm -rf /usr/obj       
 cd /usr/src
 make -j 8 buildworld     
 make -j 8 buildkernel KERNCONF=[custom kernel]
 make installworld     
 mergemaster -Ui       
 shutdown -r now
 cd /usr/src
 make check-old       
 
 and here is where I dont do anything:
 delete-old and delete-old-libs
 because I have to type 'y' and press enter for every file


taken from here(article from 2013 btw)
Building FreeBSD World And Kernel: The Short Form
 
wolffnx said:
buildworld [...] takes me too long, even with the -j switch in a intel corei7 with 8 cores
Click to expand...
Enable WITH_META_MODE in the source build environment. The first time it will take long, but subsequent builds will decrease in time.

wolffnx said:
rm -rf /usr/obj
cd /usr/src
Click to expand...
Code: 
cd /usr/src
make cleanworld
will do better. When there are previous builds, there will be immutable flags set on some files, which requires to unset them (chflags(1)), otherwise rm(1) can't empty directories containing those immuted files. "cleanworld" takes care of them (invoking chflags -R 0). See build(7) for "cleanworld" (and other build) details.

wolffnx said:
mergemaster -Ui
Click to expand...
mergemaster(8) is deprecated, use etcupdate(8).

wolffnx said:
and here is where I dont do anything:
delete-old and delete-old-libs
because I have to type 'y' and press enter for every file
Click to expand...
That can be done in batch. See my Notes below I took to upgrade to 14.0 and for future updates/upgrades:

Code: 
Read src/UPDATING.

As advised in UPDATING, COMMON ITEMS, General Notes,
"To run your build attempts in an "environmental clean room",
prefix all make commands with 'env -i '."

 - cd /usr/src-14.0
 - env -i make -jX buildworld
 - env -i make -jX buildkernel

[Major upgrade]
 - Create boot media (memstick) for emergency
 - backup system (zfs snapshot -r)

 - Edit /etc/rc.conf   /boot/loader.conf
   deactivate (comment) kernel module driver (ports, system)

[Minor/Major upgrade]
 - bectl create -r 14.0
 - bectl mount beName [mountpoint]
 - make install{kernel|world} DESTDIR=[mountpoint])

 Or
 - bectl activate 14.0
 - reboot
 - make installkernel
 - reboot into single user mode (or not)

 - etcupdate -s /usr/src-14.0 -p
   eventually
   etcupdate resolve -s /usr/src-14.0

     use "-D destdir" if bectl mount beName [mountpoint]
     after make installworld DESTDIR=
 
 - make installworld

 - etcupdate -s /usr/src-14.0 -B

     use "-D destdir" if bectl mount beName [mountpoint]

If bectl mount beName [mountpoint] reboot in upgraded beName first

 - make check-old
 - yes|make delete-old
 
   or: make -DBATCH_DELETE_OLD_FILES delete-old
 
 - make check-old-libs
 - yes|make delete-old-libs
 
   or: make -DBATCH_DELETE_OLD_FILES delete-old-libs
 
[Major Upgrade]
 - mv /boot/efi/efi/freebsd/loader.efi   loader.efi.old
   cp /boot/loader.efi /boot/efi/efi/freebsd/
 - mv /boot/efi/efi/boot/bootx64.efi   bootx64.efi.old
   cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi

 - reboot

 - zpool upgrade
 - zpool upgrade -a
 
From /sys/Makefile:

# 1. `cd /usr/src' (or to the directory containing your source tree).
# 2. `make buildworld'
# 3. `make buildkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC).
# 4. `make installkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC).
# [steps 3. & 4. can be combined by using the "kernel" target]
# 5. `reboot' (in single user mode: boot -s from the loader prompt).
# 6. `etcupdate -p'
# 7. `make installworld'
# 8. `etcupdate -B'
# 9. `make delete-old'
# 10. `reboot'
# 11. `make delete-old-libs' (in case no 3rd party program uses them anymore)

On a zfs root machine add "bectl create SOME_TAG" before doing installkernel. I typically just set it to SOME_TAG=$(date +%y%m%d), with some logic to do this only once on the same day.

You can do "yes | make delete-old" and "yes | make delete-old-libs" if you don't want to keep typing y when deleting old stuff.

You need not delete /usr/obj. That should be rarely needed if ever.
 
i do a :
1)make buildkernel
2)make installkernel
3)make buildworld
4)etcupdate -p
5) make installworld
6) etcupdate -B
7) make -DBATCH_DELETE_OLD_FILES delete-old
8) make -DBATCH_DELETE_OLD_FILES delete-old-libs
Does that looks good ?
 
I'd move step 3 before step 1. I had to do proper cleanup of 13 build remains just to get 14 to build. Don't forget to make sure packages are properly upgraded/reinstalled for the new build before delete-old-libs step. I also add steps to create backups and update boot loader and make sure I disable 3rd party kernel modules from loading on startup until rebuilt/reinstalled.
 
Thanks to all!!
this helps me a lot , for the step of "delete-old-libs" and the answer is obviuous
but is better make buildworld in a fresh install of FreeBSD , before installing the applications?
 
If you are not customizing any build options and if your source code matches the installed version then it won't matter. If it is different, then yes, programs should be installed after world is built+installed. If installing from pkg, it checks the installed base version to decide which packages to install. If installing from custom built ports then they are built against the libraries of the installed base; some ports depend on /usr/src and their build is modified even if it is not yet built/installed (best to keep /usr/src matching what is installed). If you install applications, then update source+buildworld+installworld then you should make sure pkg upgrade or rebuild ports accordingly so they don't depend on older libraries that were upgraded or removed. Once things match again, you can run the delete-old-libs step.
 
wolffnx said:
but is better make buildworld in a fresh install of FreeBSD , before installing the applications?
Click to expand...
Not going to matter. The process is pretty much 'self-contained'. I've been running the latest -STABLE versions since the 3.0 days. Always updated the OS while I had a whole bunch of ports/packages installed. Never was a problem.

Mirror176 said:
If it is different, then yes, programs should be installed after world is built+installed.
Click to expand...
Only after a major version upgrade. If you do a major version upgrade skip the 'delete-old-libs' step, do this after you have rebuild/reinstalled your installed ports/packages. The 'delete-old-libs' would delete the libraries from the previous major version (your currently installed ports/packages are linked to those).

With smaller updates on -STABLE you do need to be careful with kernel modules you installed from ports/packages. You may need to rebuild those or they might fail to load or crash the system.

bakul said:
You can do "yes | make delete-old" and "yes | make delete-old-libs" if you don't want to keep typing y when deleting old stuff.
Click to expand...
make -DBATCH_DELETE_OLD_FILES delete-old{,-libs}

Code: 
.if !defined(BATCH_DELETE_OLD_FILES)
RM_I=-i
.else
RM_I=-fv
.endif
 
I never found single user mode necessary.

That's an observation, not a recommendation … I might have used SUM during the upgrade from 14.0-CURRENT to 15.0-CURRENT, whenever that was.
 
Alain De Vos said:
Why should i "make world" before "make kernel".
Click to expand...
Because buildworld also builds the toolchain.

Code: 
     kernel-toolchain  Rebuild the tools needed for kernel compilation.  Use
                       this if you did not do a buildworld first.

Alain De Vos said:
And why two etc_update ?
Click to expand...
Because the first creates new user accounts for example, that might be needed for a successful installworld.
Code: 
     -p             Enable “pre-world” mode.  Only merge changes to files that
                    are necessary to successfully run ‘make installworld’ or
                    ‘make installkernel’.  When this flag is enabled, the
                    existing “current” and “previous” trees are left alone.
                    Instead, a temporary tree is populated with the necessary
                    files.  This temporary tree is compared against the
                    “current” tree.  This allows a normal update to be run
                    after ‘make installworld’ has completed.  Any conflicts
                    generated during a “pre-world” update should be resolved
                    by a “pre-world” resolve.
 
Once again, thanks to all , I'proper build world without problems, the only thing that never happen
(first time building world..I made it with a fresh install of bsd)
I build world,kernel, install both,make the steps and then install everything from packages
the only application with "problems" is qtbittorrent
when open close config dialog there is a delay and freeze for a moment
in console give me this:

Code: 
 OpenType support missing for "MUTT ClearlyU Wide", script 11
  OpenType support missing for "Noto Sans", script 11
  OpenType support missing for "DejaVu Sans", script 11
  OpenType support missing for "Luxi Sans", script 11
  OpenType support missing for "Nimbus Sans", script 11
  OpenType support missing for "Linux Libertine G", script 11
  OpenType support missing for "Linux Libertine Display G", script 11
  OpenType support missing for "DejaVu Serif", script 11
  OpenType support missing for "DejaVu Sans Mono", script 11
  OpenType support missing for "DejaVu Math TeX Gyre", script 11
  OpenType support missing for "Liberation Sans", script 11
  OpenType support missing for "Liberation Serif", script 11
  OpenType support missing for "Liberation Mono", script 11
  OpenType support missing for "Standard Symbols PS", script 11
  OpenType support missing for "Noto Sans Mono", script 11
  OpenType support missing for "D050000L", script 11
  OpenType support missing for "Noto Serif", script 11
  OpenType support missing for "OCR A", script 11
  OpenType support missing for "Noto Sans Symbols", script 11
  OpenType support missing for "Luxi Mono", script 11
  OpenType support missing for "Luxi Serif", script 11
  OpenType support missing for "Carlito", script 11
  OpenType support missing for "Gentium Book Basic", script 11
  OpenType support missing for "Gentium Basic", script 11
  OpenType support missing for "Linux Biolinum G", script 11
  OpenType support missing for "Caladea", script 11
  OpenType support missing for "C059", script 11
  OpenType support missing for "Estrangelo Edessa", script 11
  OpenType support missing for "Twitter Color Emoji", script 11
  OpenType support missing for "Estrangelo Nisibin Outline", script 11
  OpenType support missing for "Serto Jerusalem", script 11
  OpenType support missing for "East Syriac Ctesiphon", script 11
  OpenType support missing for "Estrangelo Quenneshrin", script 11
  OpenType support missing for "Estrangelo Talada", script 11
  OpenType support missing for "Estrangelo TurAbdin", script 11
  OpenType support missing for "Serto Urhoy", script 11
  OpenType support missing for "Serto Malankara", script 11
  OpenType support missing for "Serto Batnan", script 11
  OpenType support missing for "Estrangelo Antioch", script 11
  OpenType support missing for "East Syriac Adiabene", script 11
  OpenType support missing for "URW Gothic", script 11
  OpenType support missing for "URW Bookman", script 11
  OpenType support missing for "Serto Kharput", script 11
  OpenType support missing for "P052", script 11
  OpenType support missing for "Serto Jerusalem Outline", script 11
  OpenType support missing for "Serto Mardin", script 11
  OpenType support missing for "Nimbus Sans Narrow", script 11
  OpenType support missing for "Special Elite", script 11
  OpenType support missing for "Estrangelo Midyat", script 11
  OpenType support missing for "Classic Console", script 11
  OpenType support missing for "Nimbus Roman", script 11
  OpenType support missing for "Estrangelo Nisibin", script 11
  OpenType support missing for "Nimbus Mono PS", script 11
  OpenType support missing for "Cantarell", script 11
  OpenType support missing for "Bitstream Charter", script 11
  OpenType support missing for "Courier", script 11
  OpenType support missing for "Courier 10 Pitch", script 11
  OpenType support missing for "Utopia", script 11
  OpenType support missing for "gallant12x22", script 11
  OpenType support missing for "Goha-Tibeb Zemen", script 11
  OpenType support missing for "Cursor", script 11
  OpenType support missing for "Z003", script 11
  OpenType support missing for "Adobe Utopia", script 11
  OpenType support missing for "Adobe Courier", script 11
  OpenType support missing for "Adobe New Century Schoolbook", script 11
  OpenType support missing for "Schumacher Clean", script 11
  OpenType support missing for "B&H LucidaBright", script 11
  OpenType support missing for "Misc Fixed", script 11
  OpenType support missing for "Adobe Helvetica", script 11
  OpenType support missing for "Adobe Times", script 11
  OpenType support missing for "B&H LucidaTypewriter", script 11
  OpenType support missing for "B&H Lucida", script 11
  OpenType support missing for "Misc Fixed Wide", script 11
  OpenType support missing for "DEC Terminal", script 11
  OpenType support missing for "Sony Fixed", script 11
  OpenType support missing for "MUTT ClearlyU Alternate Glyphs Wide", script 11
  OpenType support missing for "MUTT ClearlyU PUA", script 11
  OpenType support missing for "Schumacher Clean Wide", script 11
  OpenType support missing for "Bitstream Terminal", script 11
  OpenType support missing for "micro.pcf", script 11
  OpenType support missing for "Arabic Newspaper", script 11
  OpenType support missing for "cursor.pcf", script 11
  OpenType support missing for "deccurs.pcf", script 11
  OpenType support missing for "decsess.pcf", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
 OpenType support missing for "MUTT ClearlyU Wide", script 11
  OpenType support missing for "Noto Sans", script 11
  OpenType support missing for "DejaVu Sans", script 11
  OpenType support missing for "Luxi Sans", script 11
  OpenType support missing for "Nimbus Sans", script 11
  OpenType support missing for "Linux Libertine G", script 11
  OpenType support missing for "Linux Libertine Display G", script 11
  OpenType support missing for "DejaVu Serif", script 11
  OpenType support missing for "DejaVu Sans Mono", script 11
  OpenType support missing for "DejaVu Math TeX Gyre", script 11
  OpenType support missing for "Liberation Sans", script 11
  OpenType support missing for "Liberation Serif", script 11
  OpenType support missing for "Liberation Mono", script 11
  OpenType support missing for "Standard Symbols PS", script 11
  OpenType support missing for "Noto Sans Mono", script 11
  OpenType support missing for "D050000L", script 11
  OpenType support missing for "Noto Serif", script 11
  OpenType support missing for "OCR A", script 11
  OpenType support missing for "Noto Sans Symbols", script 11
  OpenType support missing for "Luxi Mono", script 11
  OpenType support missing for "Luxi Serif", script 11
  OpenType support missing for "Carlito", script 11
  OpenType support missing for "Gentium Book Basic", script 11
  OpenType support missing for "Gentium Basic", script 11
  OpenType support missing for "Linux Biolinum G", script 11
  OpenType support missing for "Caladea", script 11
  OpenType support missing for "C059", script 11
  OpenType support missing for "Estrangelo Edessa", script 11
  OpenType support missing for "Twitter Color Emoji", script 11
  OpenType support missing for "Estrangelo Nisibin Outline", script 11
  OpenType support missing for "Serto Jerusalem", script 11
  OpenType support missing for "East Syriac Ctesiphon", script 11
  OpenType support missing for "Estrangelo Quenneshrin", script 11
  OpenType support missing for "Estrangelo Talada", script 11
  OpenType support missing for "Estrangelo TurAbdin", script 11
  OpenType support missing for "Serto Urhoy", script 11
  OpenType support missing for "Serto Malankara", script 11
  OpenType support missing for "Serto Batnan", script 11
  OpenType support missing for "Estrangelo Antioch", script 11
  OpenType support missing for "East Syriac Adiabene", script 11
  OpenType support missing for "URW Gothic", script 11
  OpenType support missing for "URW Bookman", script 11
  OpenType support missing for "Serto Kharput", script 11
  OpenType support missing for "P052", script 11
  OpenType support missing for "Serto Jerusalem Outline", script 11
  OpenType support missing for "Serto Mardin", script 11
  OpenType support missing for "Nimbus Sans Narrow", script 11
  OpenType support missing for "Special Elite", script 11
  OpenType support missing for "Estrangelo Midyat", script 11
  OpenType support missing for "Classic Console", script 11
  OpenType support missing for "Nimbus Roman", script 11
  OpenType support missing for "Estrangelo Nisibin", script 11
  OpenType support missing for "Nimbus Mono PS", script 11
  OpenType support missing for "Cantarell", script 11
  OpenType support missing for "Bitstream Charter", script 11
  OpenType support missing for "Courier", script 11
  OpenType support missing for "Courier 10 Pitch", script 11
  OpenType support missing for "Utopia", script 11
  OpenType support missing for "gallant12x22", script 11
  OpenType support missing for "Goha-Tibeb Zemen", script 11
  OpenType support missing for "Cursor", script 11
  OpenType support missing for "Z003", script 11
  OpenType support missing for "Adobe Utopia", script 11
  OpenType support missing for "Adobe Courier", script 11
  OpenType support missing for "Adobe New Century Schoolbook", script 11
  OpenType support missing for "Schumacher Clean", script 11
  OpenType support missing for "B&H LucidaBright", script 11
  OpenType support missing for "Misc Fixed", script 11
  OpenType support missing for "Adobe Helvetica", script 11
  OpenType support missing for "Adobe Times", script 11
  OpenType support missing for "B&H LucidaTypewriter", script 11
  OpenType support missing for "B&H Lucida", script 11
  OpenType support missing for "Misc Fixed Wide", script 11
  OpenType support missing for "DEC Terminal", script 11
  OpenType support missing for "Sony Fixed", script 11
  OpenType support missing for "MUTT ClearlyU Alternate Glyphs Wide", script 11
  OpenType support missing for "MUTT ClearlyU PUA", script 11
  OpenType support missing for "Schumacher Clean Wide", script 11
  OpenType support missing for "Bitstream Terminal", script 11
  OpenType support missing for "micro.pcf", script 11
  OpenType support missing for "Arabic Newspaper", script 11
  OpenType support missing for "cursor.pcf", script 11
  OpenType support missing for "deccurs.pcf", script 11
  OpenType support missing for "decsess.pcf", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11
  OpenType support missing for "", script 11

  OpenType support missing for "", script 11

but i can live with that, the other QT apps give me no problem
 
astyle said:
Uhhh... buildworld and buildkernel do take a long time, to the tune of a few hours.
Click to expand...
Buildkernel shouldn't take that long, even on a moderate system. Buildworld on the other hand, yes, that can take a while. Especially a "full" buildworld. I really like the WITH_META_MODE option, that speeds up things considerably by not rebuilding everything from scratch but only what's actually needed to be rebuild. It's not perfect but it certainly helps.

 
My scheme is so :
0.stop a jail
1.buildworld
2.buildkernel
3.installkernel
4.etcupdate -p
5.installworld
6.etcupdate -B
7.make -DBATCH_DELETE_OLD_FILES delete-old
8.make -DBATCH_DELETE_OLD_FILES delete-old-libs
9.reboot
This worked well.
 
Reasons to not combine steps into 1 make command:
  • different count of make jobs:
    • updating status.
    • world vs kernel.
    • thought I recall recommending install be single threaded (and probably still wise for mechanical drives).
  • PORTS_MODULES defined + any '-j' paramater during buildkernel with a ports config dialog opening locks up the build. workarounds exist (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259702):
    • define BATCH to avoid config dialogs.
    • preemptively run through the dialogs before this command; make sure to also get dependencies. portupgrade and portmaster have commands for that if I recall.
    • leave -j off entirely for buildkernel.
    • manually handle the PORTS_MODULES build steps outside the `make buildkernel` command
 
Before I switched to pkgbase, my routine included this:

cd /usr/src && time make -j 16 buildworld | tee /var/log/buildworld.log > /dev/null && time make DISABLE_VULNERABILITIES=yes -j 16 buildkernel | tee /var/log/buildkernel.log > /dev/null && grep -C 2 completed\ on /var/log/buildworld.log && grep -C 2 completed\ on /var/log/buildkernel.log
 
I'm not sure if disabling vulnerabilities sounds like should make the most secure kernel ever or just make the build process safe. For those not familiar, I think it is here to make sure that PORTS_MODULES build despite any port+dependency being marked vulnerable.

cd /usr/src&&git pull --ff-only&&git gc&&/usr/bin/nice -n 18 /usr/sbin/idprio 31 make -sj8 buildworld&&make -sj8 buildkernel&&make installkernel
was my last style. `git gc` is more about experimentation if that helps with zfs fragmentation of the layout of the git maintained repository + files under its watch degrading performance but I think it should be done only after a while or noticed performance impact as regularly may cause its own version of the similar problem but still testing. I do not use PORTS_MODULES, I build all ports separately with poudriere before the update and don't update the ports tree, then update poudriere jail to newly biult version (before installkernel would be the least downtime way; use extra delay to see if any new 'stuff broke' emails or patches hit mailing list/source tree just before install). idprio used to not really make a noticeable impact but recent testing shows it is quite noticeable if on an otherwise idle machine; need more testing.

I haven't been logging builds but I made an exception as 14 fails to build from a 'dirty' 13 environment until I inserted
#chflags -R noschg /usr/obj/usr;rm -rf /usr/obj/usr;cd /usr/src&&make cleandir&&make cleandir
which I thought I'd test again someday and ask mailing list if information about that is relevant; I remember that being a recommended step somewhere but haven't ran across it in documentation lately. I have been using meta mode and don't remember if I tested if that impacted the failure.

Someday I will add better testing than '&&' aborting future commands so I can make my build scripts more reliable and optional in their actions but when it works good enough without issue its hard to care.

Since moving to git, anyone know if `make update` is supported or deprecated? I remember it being a thing in cvs and svn days where make.conf had variables for servers and settings to get the update completed.
 
I have a script that I run once a day (more or less) that builds current, stable/14 and stable/13. I build current for arm64 as well. Another script lets me review git log since last commit these three branches were built locally, to keep an eye on what is changing. Anyway here is a brief log:
# mkw curr 14 13 -A aarch64 curr &
### when it is done:
# grep 'built in' ~/a*64/*/err
/root/aarch64/current/err:>>> World built in 119 seconds, ncpu: 16, make -j18
/root/aarch64/current/err:>>> Kernel(s) GENERIC-NODEBUG built in 244 seconds, ncpu: 16, make -j18
/root/amd64/current/err:>>> World built in 129 seconds, ncpu: 16, make -j18
/root/amd64/current/err:>>> Kernel(s) GENERIC built in 300 seconds, ncpu: 16, make -j18
/root/amd64/stable/err:>>> World built in 89 seconds, ncpu: 16, make -j18
/root/amd64/stable/err:>>> Kernel(s) GENERIC built in 114 seconds, ncpu: 16, make -j18
/root/amd64/stable13/err:>>> World built in 135 seconds, ncpu: 16, make -j18
/root/amd64/stable13/err:>>> Kernel(s) GENERIC built in 48 seconds, ncpu: 16, make -j18
This build machine is a 5 year old Ryzen machine, with WD "Gold" disks.

For every successful build I install kernel, world on at least the VMs & pi4 but may also do so x86-64 machines (all except the build machine, which gets updated about monthly).
 
You must log in or register to reply here.
Back
Top