14.0-RELEASE weird build world/kernel/ports behavior

epopen

epopen

Hi All

I upgraded from 13.2-RELEASE to 14.0-RELAESE by source code and one-shot build successful (git branch: releng/14.0)
H/W: 16GB Physical RAM + 10 GB SWAP @ Root on ZFS
But encounter build world/kernel/ports terminate anywhere randomly even code have no problem.:eek:
  • Example (buildworld)
    Code: 
    .PATH='. /usr/src/lib/libc /usr/src/lib/libc/csu /usr/src/lib/libc/db/btree /usr/src/lib/libc/db/db /usr/src/lib/libc/db/hash /usr/src/lib/libc/db/man /usr/src/lib/libc/db/mpool /usr/src/lib/libc/db/recno
/usr/src/lib/libc/compat-43 /usr/src/lib/libc/gdtoa /usr/src/lib/libc/amd64/gen /usr/src/lib/libc/gen /usr/src/etc /usr/src/contrib/libc-pwcache /usr/src/contrib/libc-vis /usr/src/lib/libc/gmon
/usr/src/lib/libc/iconv /usr/src/lib/libc/inet /usr/src/lib/libc/isc /usr/src/lib/libc/locale /usr/src/lib/libmd /usr/src/lib/libc/nameser /usr/src/lib/libc/net /usr/src/lib/libc/nls /usr/src/lib/libc/posix1e
/usr/src/lib/libc/regex /usr/src/lib/libc/resolv /usr/src/lib/libc/stdio /usr/src/lib/libc/amd64/stdlib /usr/src/lib/libc/stdlib /usr/src/lib/libc/stdtime /usr/src/contrib/tzcode /usr/src/lib/libc/amd64/string
/usr/src/lib/libc/string /usr/src/sys/libkern /usr/src/lib/libc/amd64/sys /usr/src/lib/libc/sys /usr/src/lib/libc/secure /usr/src/lib/libc/rpc /usr/src/lib/libc/. /usr/src/lib/libc/uuid /usr/src/lib/libc/xdr
/usr/src/lib/libc/x86/sys /usr/src/lib/libc/x86/gen /usr/src/sys/kern /usr/src/lib/libc/capability'
ERROR: ctfconvert: failed to initialize DWARF: (null) [_dwarf_elf_decompress(235)]
    Stall over 1 hour until I Ctrl+C, but cc running in 1 hour still as follows
    Code: 
    92139 root          1 135    0   114M    63M CPU0     0  45:40 100.11% cc
  • Example (buildworld)
    Code: 
    Building /usr/obj/usr/src/amd64.amd64/lib/libc/putwchar.o
Segmentation fault
*** [puts.o] Error code 139
make[4]: *** puts.o removed
  • Example (buildworld)
    Code: 
    Building /usr/obj/usr/src/amd64.amd64/lib/msun/e_powl.o
ERROR: ctfconvert: failed to initialize DWARF: (null) [_dwarf_elf_decompress(235)]
Building /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto/a_object.o
PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /usr/bin/cc -target x86_64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -fno-common -march=native -ffp-exc
eption-behavior=maytrap -fno-math-errno -g -gz=zlib -std=gnu99 -Wno-format-zero-length -fstack-protector-strong -Wno-error=ignored-pragmas -Wsystem-headers -Wno-pointer-sign -Wdate-time -Wno-unknown-pragm
as -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-parameter -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-
conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -Werror -I/usr/src/lib/msun/x86 -I/usr/src/l
ib/msun/ld80 -I/usr/src/lib/msun/amd64 -I/usr/src/lib/msun/src -I/usr/src/lib/libc/include -I/usr/src/lib/libc/amd64 -c -o e_lgammal_r.o /usr/src/lib/msun/ld80/e_lgammal_r.c
1.      /usr/src/lib/msun/ld80/e_lgammal_r.c:96:68: current parser token ','
 #0 0x0000000004f8d411 PrintStackTrace /usr/src/contrib/llvm-project/llvm/lib/Support/Unix/Signals.inc:567:13
 #1 0x0000000004f8b825 RunSignalHandlers /usr/src/contrib/llvm-project/llvm/lib/Support/Signals.cpp:105:18
 #2 0x0000000004f33ec5 HandleCrash /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:73:5
 #3 0x0000000004f33ec5 CrashRecoverySignalHandler /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:390:51
...
#39 0x0000000002cff290 ExecuteCompilation /usr/src/contrib/llvm-project/clang/lib/Driver/Driver.cpp:1856:23
#40 0x00000000024762ad main /usr/src/contrib/llvm-project/clang/tools/driver/driver.cpp:562:21
#41 0x000000082e5ff63a __libc_start1 /usr/src/lib/libc/csu/libc_start1.c:157:2
cc: error: clang frontend command failed with exit code 139 (use -v to see invocation)
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Target: x86_64-unknown-freebsd14.0
Thread model: posix
InstalledDir: /usr/bin
cc: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
cc: note: diagnostic msg: /tmp/e_lgammal_r-0ffc85.c
cc: note: diagnostic msg: /tmp/e_lgammal_r-0ffc85.sh
cc: note: diagnostic msg:

********************
[e_lgammal_r.o] Error code 1
  • Example (buildworld)
    Code: 
    /usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:169:13: note: insert '_Nullable' if the pointer may be null
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:169:13: note: insert '_Nonnull' if the pointer should never be null
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:170:13: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or _Null_unspecified) [-Werror,-Wnullability-completeness]
extern FILE *__stderrp;
            ^
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:170:13: note: insert '_Nullable' if the pointer may be null
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:170:13: note: insert '_Nonnull' if the pointer should never be null
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
*** [h_hash.o] Error code 1

make[7]: stopped in /usr/src/lib/libc/tests/hash
  • Example (buildworld)
    Build procedure terminated but cc running still
    😳

    Screenshot from 2023-11-23 10-55-18.png
  • Example (buildworld)
    Stall over 30 minutes.
    Screenshot from 2023-11-23 11-22-21.png
  • Example (ports mysql80-server)

    Code: 
    [ 76%] Building CXX object sql/CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o
cd /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/sql && /usr/local/libexec/ccache/c++ -DBOOST_NO_CXX98_FUNCTION_BASE -DHAVE_CONFIG_H -DLZ4_DISABLE_DEPRECATE_WARNINGS -DMYSQL_SERVER -DRAP
IDJSON_NO_SIZETYPEDEFINE -DRAPIDJSON_SCHEMA_USE_INTERNALREGEX=0 -DRAPIDJSON_SCHEMA_USE_STDREGEX=1 -D_USE_MATH_DEFINES -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/PortsWRKDIR/usr/ports/databases/my
sql80-server/work/.build -I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/include -I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35 -I/usr/PortsWRKDIR/usr/ports/data
bases/mysql80-server/work/mysql-8.0.35/include -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/protobuf/protobuf-3.19.4/src -isystem /usr/local/include/editline -isyst
em /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/include/boost_1_77_0/patches -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/boost/boost_1_77_0 -isy
stem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/zlib/zlib-1.2.13 -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/extra/zlib/zlib-1.2.13 -isystem /
usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/rapidjson/include -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/zstd/zstd-1.5.5/lib -isyst
em /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/plugin/x/client -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/plugin/x/generated/protobuf_lite -isystem
/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/plugin/x/generated -std=c++17 -fno-omit-frame-pointer -ftls-model=initial-exec -O2 -pipe -march=native  -fPIC -fstack-protector-strong -isys
tem /usr/local/include -fno-strict-aliasing   -malign-double -isystem /usr/local/include -std=c++17 -Wall -Wextra -Wformat-security -Wvla -Wundef -Wmissing-format-attribute -Woverloaded-virtual -Wcast-qua
l -Wno-null-conversion -Wno-unused-private-field -Wconditional-uninitialized -Wdeprecated -Wextra-semi -Wheader-hygiene -Wnon-virtual-dtor -Wundefined-reinterpret-cast -Wrange-loop-analysis -Winconsistent
-missing-destructor-override -Winconsistent-missing-override -Wshadow-field -ffunction-sections -fdata-sections -O2 -pipe -march=native  -fPIC -fstack-protector-strong -isystem /usr/local/include -fno-str
ict-aliasing   -malign-double -isystem /usr/local/include -std=c++17  -DNDEBUG -fPIC -Wshadow-uncaptured-local -MD -MT sql/CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o -MF CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o.d
 -o CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o -c /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/sql/dd/impl/dd.cc
PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /usr/bin/c++ -std=c++17 -fno-omit-frame-pointer -ftls-model=initial-exec -O2 -pipe -march=native -fPIC -fstack-protector-strong -fno-strict-aliasing -malign-double -std=c++17 -W
all -Wextra -Wformat-security -Wvla -Wundef -Wmissing-format-attribute -Woverloaded-virtual -Wcast-qual -Wno-null-conversion -Wno-unused-private-field -Wconditional-uninitialized -Wdeprecated -Wextra-semi
 -Wheader-hygiene -Wnon-virtual-dtor -Wundefined-reinterpret-cast -Wrange-loop-analysis -Winconsistent-missing-destructor-override -Winconsistent-missing-override -Wshadow-field -ffunction-sections -fdata
-sections -O2 -pipe -march=native -fPIC -fstack-protector-strong -fno-strict-aliasing -malign-double -std=c++17 -fPIC -Wshadow-uncaptured-local -DBOOST_NO_CXX98_FUNCTION_BASE -DHAVE_CONFIG_H -DLZ4_DISABLE
_DEPRECATE_WARNINGS -DMYSQL_SERVER -DRAPIDJSON_NO_SIZETYPEDEFINE -DRAPIDJSON_SCHEMA_USE_INTERNALREGEX=0 -DRAPIDJSON_SCHEMA_USE_STDREGEX=1 -D_USE_MATH_DEFINES -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -
I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build -I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/include -I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-
8.0.35 -I/usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/include -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/protobuf/protobuf-3.19.4/src -is
ystem /usr/local/include/editline -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/include/boost_1_77_0/patches -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/w
ork/mysql-8.0.35/boost/boost_1_77_0 -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/zlib/zlib-1.2.13 -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/
.build/extra/zlib/zlib-1.2.13 -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/extra/rapidjson/include -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql
-8.0.35/extra/zstd/zstd-1.5.5/lib -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/mysql-8.0.35/plugin/x/client -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/pl
ugin/x/generated/protobuf_lite -isystem /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build/plugin/x/generated -isystem /usr/local/include -isystem /usr/local/include -isystem /usr/local/inclu
de -isystem /usr/local/include -DNDEBUG -c -MD -MT sql/CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o -MF CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o.d -o CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o /usr/PortsWRKDIR/usr/ports
/databases/mysql80-server/work/mysql-8.0.35/sql/dd/impl/dd.cc
1.      /usr/include/c++/v1/__type_traits/aligned_storage.h:128:1 <Spelling=/usr/include/c++/v1/__type_traits/aligned_storage.h:111:49>: current parser token ';'
2.      /usr/include/c++/v1/__type_traits/aligned_storage.h:23:1 <Spelling=/usr/include/c++/v1/__config:663:39>: parsing namespace 'std'
3.      /usr/include/c++/v1/__type_traits/aligned_storage.h:23:1 <Spelling=/usr/include/c++/v1/__config:663:62>: parsing namespace 'std::__1'
4.      /usr/include/c++/v1/__type_traits/aligned_storage.h:128:1 <Spelling=/usr/include/c++/v1/__type_traits/aligned_storage.h:106:23>: parsing struct/union/class body 'std::aligned_storage<_Len, 8192>'
5.      /usr/include/c++/v1/__type_traits/aligned_storage.h:128:1 <Spelling=/usr/include/c++/v1/__type_traits/aligned_storage.h:109:5>: parsing struct/union/class body 'std::aligned_storage<_Len, 8192>::t
ype'
 #0 0x0000000004fe5821 PrintStackTrace /usr/src/contrib/llvm-project/llvm/lib/Support/Unix/Signals.inc:567:13
 #1 0x0000000004fe3c17 RunSignalHandlers /usr/src/contrib/llvm-project/llvm/lib/Support/Signals.cpp:105:18
 #2 0x0000000004f8b435 HandleCrash /usr/src/contrib/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:73:5
...
#77 0x0000000002d115d0 ExecuteCompilation /usr/src/contrib/llvm-project/clang/lib/Driver/Driver.cpp:1856:23
#78 0x000000000247bd33 main /usr/src/contrib/llvm-project/clang/tools/driver/driver.cpp:562:21
#79 0x000000082b8fab3a __libc_start1 /usr/src/lib/libc/csu/libc_start1.c:157:2
c++: error: clang frontend command failed with exit code 139 (use -v to see invocation)
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Target: x86_64-unknown-freebsd14.0
Thread model: posix
InstalledDir: /usr/bin
c++: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
c++: note: diagnostic msg: /tmp/dd-e7c666.cpp
c++: note: diagnostic msg: /tmp/dd-e7c666.sh
c++: note: diagnostic msg:

********************
*** [sql/CMakeFiles/sql_dd.dir/dd/impl/dd.cc.o] Error code 1

make[4]: stopped in /usr/PortsWRKDIR/usr/ports/databases/mysql80-server/work/.build
1 error

I assume above 1st build system defect existence @ 13.2-RELEASE, therefore 2nd build world/kernel @ 14.0-RELEASE
I could not count how much times to restart make -j4 buildworld in 2 days because terminated randomlyo_O
But problem still after installation.:'‑(

Sorry example too much, have any suggestation/debug help me.:)
Thanks all very much.
 
Well, I'm not an FreeBSD engineer/developer (I'm not an engineer at all), but what follows is what I seen on my system. This story is very very very long, but to avoid boring you I make it simple.

One day, I don't remember when (I've neglected my system a little too much over the last six years), the ports started to give me troubles, expecially the big ones (mysql, LLVMs, and few others), then four mounths ago I decided to resolve this. I left my system at version 10.3-RELEASE, not sure if I had upgraded it at 11.0-RELEASE, probably I skipped version 11, and after about three years, I upgraded at version 12.x-RELEASE, From that release, I always upgraded it in the right way (not compile from source, always binaries with freebsd-update).

I thought I made a mistake upgrading it up to 13.2-RELEASE, so I never look at the right place to solve the problem. I guessed it was the swap space too small, when the first time I intalled the system (perhaps 9.x-RELEASE or 10.0-RELEASE) I made the swap space larger as same as the RAM (4GB). However there also was a problem with HD: running fsck -F (in multi user mode only, in single user mode give me no errors) give me some unreferenced nodes (this is only a small part of the result)

Code: 
...

UNREF FILE I=34108996  OWNER=root MODE=100600
SIZE=0 MTIME=Nov 23 18:25 2023

CLEAR? no

UNREF FILE I=34269327  OWNER=mysql MODE=100600
SIZE=0 MTIME=Nov 23 18:25 2023

CLEAR? no

UNREF FILE I=34269328  OWNER=mysql MODE=100600
SIZE=0 MTIME=Nov 23 18:25 2023

...

then I attached monitor and keyboard to the server and spent some time to try to recover that disk errors. During one reboot (after a lot of reboots in single user/multi user mode) I saw this line:

Code: 
module_register_init: MOD_LOAD (tmpfs, 0xffffffff80cd4530, 0xffffffff82148b40) error 17

then I entirely removed tmpfs from loader.conf and fstab.

This week-end I installed a bunch of ports with no clang errors. However I still have disk errors, but only in multi user mode. Hope this help you.
 
Hi All

Test update...
  1. About H/W stability, I had tested by memtest86, pi and unix bench a all day @ i5-7200U, all passed.:)
  2. Storage status of # zpool status
    Code: 
      pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:29:07 with 0 errors on Fri Nov 24 15:04:57 2023
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors
    All fine.
  3. About CPUTYPE @ /etc/make.conf
    Because fix previous issue, H/W platform change from Celeron to i5-7200U, and the issue occurred by CPUTYPE?=native (for Celeron)
    I assume root cause is incorrect model of CPUTYPE, therefore I changed to CPUTYPE?=skylake (for i5-7200U, it is Kaby Lake, but have not in system, use skylake instead)
    After clean build ( rm -rf /usr/obj/usr) and install world/kernel.. issue still o_O
Thanks a lot.
 
Hi All

Before, encounter out of swap -> system silent dead issue @ buildworld,:eek:
System statistics weird also :rolleyes:
Today, I saw the issue before system silent dead, therefore screen capture for reference.

Note:
  1. After capture, I break buildworld by Crtl+C to prevent system silent dead (100% Inuse).😅
  2. The issue occurred either customer build kernel or GENERIC @ 14-RELEASE kernel (/boot/kernel copied from USB installation media).

Out_of_swap.png
 
You must log in or register to reply here.
Back
Top