Make buildworld failing

All,

I am trying to update my server kernel from 13.2-RELEASE-p9 to 13.4-RELEASE-p2 and have a problem building the world to begin the process. The process fails very quickly..
Code:
.
.
c++  -O2 -pipe -fno-common -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm-project/llvm/include -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DHAVE_VCS_VERSION_INC -DNDEBUG -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DDEFAULT_SYSROOT=\"/usr/obj/usr/src/amd64.amd64/tmp\" -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -DNDEBUG -MD -MF.depend.Demangle_ItaniumDemangle.o -MTDemangle/ItaniumDemangle.o -Wno-format-zero-length -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include  -fno-exceptions -fno-rtti -std=c++17    -stdlib=libc++ -Wno-c++11-extensions   -c /usr/src/contrib/llvm-project/llvm/lib/Demangle/ItaniumDemangle.cpp -o Demangle/ItaniumDemangle.o
In file included from /usr/src/contrib/llvm-project/llvm/lib/Demangle/ItaniumDemangle.cpp:13:
In file included from /usr/src/contrib/llvm-project/llvm/include/llvm/Demangle/Demangle.h:12:
/usr/include/c++/v1/cstddef:45:10: fatal error: '__nullptr' file not found
#include <__nullptr>
         ^~~~~~~~~~~
1 error generated.
*** Error code 1

Stop.
make[3]: stopped in /usr/src/lib/clang/libllvmminimal
*** Error code 1

Stop.
make[2]: stopped in /usr/src
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

I have previously updated this server with the following settings in /etc/src.conf (trying to keep things small as I have limited disk space)

Code:
# Specify my VM kernel config
KERNCONF=VM

# Kernel modules - none please?
MODULES_OVERRIDE = geom aesni

# Delete all old files
BATCH_DELETE_OLD_FILES=YES

# make world tuning
WITHOUT_AMD=YES
WITHOUT_APM=YES
WITHOUT_ASSERT_DEBUG=YES
WITHOUT_ATM=YES
WITHOUT_AUTHPF=YES
WITHOUT_AUTOFS=YES
WITHOUT_BHYVE=YES
WITHOUT_BLUETOOTH=YES
WITHOUT_BOOTPARAMD=YES
WITHOUT_BOOTPD=YES
WITHOUT_BSDINSTALL=YES
WITHOUT_BSNMP=YES
WITHOUT_CCD=YES
WITHOUT_CDDL=YES
WITHOUT_CLANG_EXTRAS=YES
WITHOUT_CLANG_FULL=YES
WITHOUT_CROSS_COMPILER=YES
WITHOUT_CTF=YES
WITHOUT_CTM=YES
WITHOUT_CXGBETOOL=YES
WITHOUT_DEBUG_FILES=YES
WITHOUT_DTRACE_TESTS=YES
WITHOUT_EE=YES
WITHOUT_EXAMPLES=YES
WITHOUT_FINGER=YES
WITHOUT_FLOPPY=YES
WITHOUT_FREEBSD_UPDATE=YES
WITHOUT_GAMES=YES
WITHOUT_GSSAPI=YES
WITHOUT_HAST=YES
WITHOUT_HTML=YES
WITHOUT_HYPERV=YES
WITHOUT_IPFILTER=YES
WITHOUT_IPFW=YES
WITHOUT_ISCSI=YES
WITHOUT_JAIL=YES
WITHOUT_KERBEROS=YES
WITHOUT_KERBEROS_SUPPORT=YES
WITHOUT_KERNEL_SYMBOLS=YES
WITHOUT_LLDB=YES
WITHOUT_LPR=YES
WITHOUT_MLX5TOOL=YES
WITHOUT_NDIS=YES
WITHOUT_NETGRAPH=YES
WITHOUT_NIS=YES
WITHOUT_PC_SYSINSTALL=YES
WITHOUT_PMC=YES
WITHOUT_PORTSNAP=YES
WITHOUT_PPP=YES
WITHOUT_PROFILE=YES
WITHOUT_QUOTAS=YES
WITHOUT_RADIUS_SUPPORT=YES
WITHOUT_RBOOTD=YES
WITHOUT_RCMDS=YES
WITHOUT_ROUTED=YES
WITHOUT_SOURCELESS=YES
WITHOUT_SOURCELESS_HOST=YES
WITHOUT_SOURCELESS_UCODE=YES
WITHOUT_TALK=YES
WITHOUT_TESTS=YES
WITHOUT_TESTS_SUPPORT=YES
WITHOUT_TFTP=YES
WITHOUT_TIMED=YES
WITHOUT_WIRELESS=YES
WITHOUT_WIRELESS_SUPPORT=YES
WITHOUT_WPA_SUPPLICANT_EAPOL=YES
WITHOUT_ZFS=YES

So, I checked if I could compile a port and YES, I can make git without an issue.

I then tried commenting out all the WITHOUT_ lines in /etc/src.conf but it fails at the same point.

I am wondering if I have previously made an error when updating which did not show until I went to update once again.

On a hunch I compared the kernel date with the date of the include file in the error trace..
Code:
# ll /boot/kernel/kernel
-r-xr-xr-x  1 root  wheel  13830160 Dec 28  2023 /boot/kernel/kernel*

 # ll /usr/include/c++/v1/cstddef
-r--r--r--  1 root  wheel  5537 Apr  7  2023 /usr/include/c++/v1/cstddef

I'm not sure if I'm on the right track or not.

Can someone please point me in the right direction?

Many thanks for all assistance!
 
Many thanks skeletor, I make progress! The compile now continues but fails at the linker

Code:
c++ -O2 -pipe -fno-common -I/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvm -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm-project/llvm/include -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DHAVE_VCS_VERSION_INC -DNDEBUG -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DDEFAULT_SYSROOT=\"/usr/obj/usr/src/amd64.amd64/tmp\" -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -DNDEBUG -Wno-format-zero-length -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -fno-exceptions -fno-rtti -std=c++17 -stdlib=libc++ -Wno-c++11-extensions  -Wl,--gc-sections -Wl,-zrelro -static   -L/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib -o llvm-min-tblgen  TableGen.o VTEmitter.o /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libexecinfo -lexecinfo -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libelf -lelf -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/ncurses/ncurses -lncursesw -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libthr -lpthread  -legacy
ld: error: undefined symbol: std::__1::__libcpp_verbose_abort(char const*, ...)
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__1::__throw_length_error[abi:sn180100](char const*)) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__throw_bad_array_new_length[abi:sn180100]()) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__1::__throw_bad_function_call[abi:sn180100]()) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced 2 more times
c++: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[3]: stopped in /usr/src/usr.bin/clang/llvm-min-tblgen
*** Error code 1

I am guessing a std library is missing or similar. I have noticed in /usr/lib dates are not consistent, whereas on a known good machine they are..

Code:
-r--r--r--  1 root  wheel  2520718 Oct 24  2023 libc++.a
-r--r--r--  1 root  wheel       68 Apr  7  2023 libc++.so
-r--r--r--  1 root  wheel  1028992 Dec 28  2023 libc++.so.1

Is this relevant?

Many thanks.
 
Try to clean all previous builds kernel and world and rebuild kernel and world with fix in my previous post.
This error indicates some clang problems and I don't have any ideas to show the right way to solve it.
Could you try build with 14.1 Release? Does it still contain error?
 
Many thanks skeletor, I make progress! The compile now continues but fails at the linker

Code:
c++ -O2 -pipe -fno-common -I/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvm -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm-project/llvm/include -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DHAVE_VCS_VERSION_INC -DNDEBUG -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd13.4\" -DDEFAULT_SYSROOT=\"/usr/obj/usr/src/amd64.amd64/tmp\" -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -DNDEBUG -Wno-format-zero-length -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -fno-exceptions -fno-rtti -std=c++17 -stdlib=libc++ -Wno-c++11-extensions  -Wl,--gc-sections -Wl,-zrelro -static   -L/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib -o llvm-min-tblgen  TableGen.o VTEmitter.o /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libexecinfo -lexecinfo -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libelf -lelf -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/ncurses/ncurses -lncursesw -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libthr -lpthread  -legacy
ld: error: undefined symbol: std::__1::__libcpp_verbose_abort(char const*, ...)
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__1::__throw_length_error[abi:sn180100](char const*)) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__throw_bad_array_new_length[abi:sn180100]()) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced by CommandLine.cpp
>>>               CommandLine.o:(std::__1::__throw_bad_function_call[abi:sn180100]()) in archive /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
>>> referenced 2 more times
c++: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[3]: stopped in /usr/src/usr.bin/clang/llvm-min-tblgen
*** Error code 1

I am guessing a std library is missing or similar. I have noticed in /usr/lib dates are not consistent, whereas on a known good machine they are..

Code:
-r--r--r--  1 root  wheel  2520718 Oct 24  2023 libc++.a
-r--r--r--  1 root  wheel       68 Apr  7  2023 libc++.so
-r--r--r--  1 root  wheel  1028992 Dec 28  2023 libc++.so.1

Is this relevant?

Many thanks.

You need to remove stale C++ include files.
 
Many thanks cracauer@ and my apologies for the delay is continuing the thread. Stale C++ include files makes sense to me but perhaps I need a little more guidance on my next steps.

So, I believe you are talking about header files in /usr/include and yes when I look in the directory I see a mixture of dates. My kernel has a date of 2023/12/28 but I can see many files in the include directory are earlier
Code:
-r--r--r--   1 root  wheel    2649 Apr  7  2023 stddef.h
lrwxr-xr-x   1 root  wheel      12 Dec 28  2023 stdint.h@ -> sys/stdint.h
-r--r--r--   1 root  wheel   17326 Apr  7  2023 stdio.h
-r--r--r--   1 root  wheel   11501 Apr  7  2023 stdlib.h
-r--r--r--   1 root  wheel    1626 Apr  7  2023 stdnoreturn.h
-r--r--r--   1 root  wheel    5592 Apr  7  2023 string.h

Comparing to a machine that did upgrade properly I see the kernel has a date of 29th Nov and the files in /usr/include are again and earlier date.

Code:
-r--r--r--   1 root  wheel   17313 Oct 12 23:31 stdio.h
-r--r--r--   1 root  wheel   11488 Oct 12 23:31 stdlib.h

Apologies, are these the include files you mention? If so, which ones should I delete/move? From the above error trace I'm guessing at least stdlib.h but they appear to be very similar (both from 1995!) but the version on the broken box appears to have an additional comment line at the top..

Code:
 * $FreeBSD$

Thanks again for responding to my post. I feel I am close to a solution . Not sure how this 'broke' however.
 
I read somewhere that this is also recommended.
make delete-old-libs
This is quite, quite and quite DANGEROUS operation when appropriate, unlike make delete-old.
Beware NO PORTS/PKGS INSTALLED ARE LINKED AGAINST THE LIBRARIES TO BE DELETED!
Otherwise the appropriate ports/pkgs SHOULD CERTAINLY stop working.
Base are fine, though, unless the committer marked the libraries as old didn't overllook anything.
 
On major version upgrade (i.e., 13.* to 14.*), misc/compat*x like misc/compat13x usually exists. So, in case upgrading from 13.* to 14.2, in many case, misc/compat13x would help after make delete-old-libs, but it's NOT ALWAYS assured to work, as in some cases it doesn't work, because of version bumps within 13.* timespan.
This sometimes happen on base LLVM/Clang upgrades and/or some contributed external components.
 
misc/compat13x is different from misc/compat12x etc.
misc/compat13x only has OpenSSL's libcrypto and libssl.
Moreover, it was created after 14.0R was released.
I looked in /usr/src/ObsoleteFiles.inc and copied the libraries from 13.x to /usr/local/lib/compat/.
There are over 30 files.
If I recall correctly, initially these misc/compat*x was for libc. And increase its contents as time goes by.
And as upgrading fromlatest minor version of previous (installed) major version to its next major version only is "officially" supported, deleted libraries at older minor version (point releases) would not be included, unless it is clearly hazardous (like OpenSSL). Just my guess.

Recently, some newer libraries would (technically) be able to support older ABI as of symbol versioning. So if the newer version of deleted old version supports symbol versioning, it would not be included in misc/compat*x for it.
 
For example, pam_opie.so.6 does not exist in either 14.x or compat13x.

I believe the reason it mentions compat12x here instead of compat13x is because compat13x is different from compat12x and earlier.
 
For example, pam_opie.so.6 does not exist in either 14.x or compat13x.
Does it exist on latest patch release of 13.4 (13.4-Release-p*)?
If so, it should be included in misc/compat13x unless it's deleted with security reason.(I'm not the maintainer, though). And pam_* libraries are for authentication, means, security-related libraries.

And deletion of opie from is mentioned on RELNOTES. with workaround (install security/opie, not misc/compat13x).
Usually, Release Engineering Team or relevant committer adds RELNOTE entries (and/or UPDATING entries) for this kind of things.
 
It was present in 13.2R when compat13x was created.
There were also some pkgs in 13.2R that linked.
They will not work even if you install compat13x.
This is an example where compat13x is different from compat12x and earlier.
I don't want to discuss opie.
 
It was present in 13.2R when compat13x was created.
There were also some pkgs in 13.2R that linked.
They will not work even if you install compat13x.
This is an example where compat13x is different from compat12x and earlier.
I don't want to discuss opie.
So you'd better file a bug for it. (ports->individual ports, then starting summary with misc/compat13x:.)
 
The 13.x ports-mgmt/dialog4ports pkg does not work because it does not have libdialog.so.9.

I'm not complaining about compat13x.
I am referring to what you wrote.
Reporting bugs is a whole different story.

> So, in case upgrading from 13.* to 14.2, in many case, misc/compat13x would help after make delete-old-libs,
When you wrote this, were you aware that compat13x was different from compat12x and earlier?
Edit: It wasn't something I'd post multiple times. It's not good to have too much free time.
Edit: commit log
> misc/compat13x: Add port.
> This port provides only base OpenSSL 1.1.1 for now
 
This is quite, quite and quite DANGEROUS operation when appropriate, unlike make delete-old.
Beware NO PORTS/PKGS INSTALLED ARE LINKED AGAINST THE LIBRARIES TO BE DELETED!
Otherwise the appropriate ports/pkgs SHOULD CERTAINLY stop working.
Base are fine, though, unless the committer marked the libraries as old didn't overllook anything.
https://forums.freebsd.org/threads/make-buildworld-failed-while-upgrading-14-stable.96051/
post #1.
Why weren't there any comments here?
Where could the topic starter get these instructions?
I'm currently "buildworld" (v.15.0-CURRENT) on my stand using these recommendations. Honestly, I don't have any software installed from ports or binaries yet, except for git ( pkg git install).
Maybe that's the difference: if there's no software, then you can and there won't be any consequences?
 
I've not comment there the same as the OP stated that OP ALREADY DID make delete-old-libs on #1. Already too late.

And the more strong warning and instructions for make delete-old-libs should be described in UPDATING.
As even /usr/local/sbin/pkg could be affected, in the cases like OpenSSL libraries bump. For those cases, /usr/local/sbin/pkg-static should basically work, as it is linked against required-to-work libraries statically (means, contained in itself).

To check whether anything in subdirectories of /usr/local/ for binary programs are linked against specified-in-the-script libraries, I'm using attached exapmle (copy, edit and run as root, then run the generated script as root to rebuild all found ports, re-run the script to confirm nothing remains for rebuild). Comments are codes that are not needed in this specific case.

To confirm which libraries to specify, you can run make check-old-libs to omit non-library old things (regular files, directories).
 

Attachments

Thanks to all for your contributions. I have also been doing some more research and I believe you are all 'on the money' in various forms.

I now believe my /usr/include/c++/v1 and /usr/lib/clang/14.0.5 directories got zapped when I did a make delete-old and -libs before I started the compilation. I eventually found that by carefully replacing the missing files from cgit I could finally rebuild 13.2 without failure. Note that I had to replace the files from the same release version to be consistent and work correctly.

Why 'live' files were wrongly identified as no longer needed I still don't know, but I can confirm that it has happened to me on two different machines running 13.2-RELEASE-p9 so far.

In my case I am trying to go from release 13.2 to 13.4 and so once I successfully compiled 13.2 I tried and failed at building 13.4! I thought some more and then moved to build 13.3, which built successfully! My next move will be to try to go from 13.3 to 13.4.

I hope this is useful..
 
To check whether anything in subdirectories of /usr/local/ for binary programs are linked against specified-in-the-script libraries, I'm using attached exapmle (copy, edit and run as root, then run the generated script as root to rebuild all found ports, re-run the script to confirm nothing remains for rebuild). Comments are codes that are not needed in this specific case.
Thank you so much, T-Aoki! Very professional approach.
I hope this is useful..
Yes. There is much to learn from knowledgeable people. The legacy of the forum is impressive.
 
The final chapter. Yes, the upgrade from 13.3 to 13.4 went without issue, on two different machines. So, lessons learned include being mindful of make delete-old and -libs and going one minor version at a time. I'll do some more research about what it deletes and why.
 
Back
Top