Rust does not build anymore

ssbear

ssbear

Hi folks.

It's been one week I can't build lang/rust anymore on my FreeBSD poudriere. However, port has not been updated for few weeks... I have to re-build again it, probably for a dependency. Fail appears after 7 hours. I try (again) this morning.Here the last lines of my log:
Code: 
cmake/crt-config-ix.cmake"
Copy/Link "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/config.toml" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/builder-config"
Install "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/LICENSE.TXT" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/LICENSE.TXT"
Copy/Link "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/LICENSE.TXT" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/LICENSE.TXT"
Install "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/README.txt" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/README.txt"
Copy/Link "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/README.txt" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/README.txt"
  c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false } }
  c RustInstaller { compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false }, target: x86_64-unknown-freebsd }
  c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false } }
Dist rust-dev-1.90.0-x86_64-unknown-freebsd
Command has failed. Rerun with -v to see more details.
Traceback (most recent call last):
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/x.py", line 53, in <module>
    bootstrap.main()
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 1377, in main
    bootstrap(args)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 1352, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 237, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /wrkdirs/usr/ports/lang/rust/work/_build/bootstrap/debug/bootstrap dist --jobs=1
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/rust
=>> Cleaning up wrkdir
===>  Cleaning for rust-1.90.0_1
build of lang/rust | rust-1.90.0_1 ended at Mon Nov 10 16:13:08 CET 2025
build time: 07:13:15
!!! build failure encountered !!!

My memory seems to be OK, and swap was used until 33% at the end of the build (obviously with my 16 Go of RAM, I need a swap to build this pain-in-the-ass port). I use FreeBSD 14.3-RELEASE-p5 on amd64 arch.

I've reseted options for lang/rust, I've also deleted my 143Ramd64-options (so, options are on default values). I also tried to drop WASM option (see this bug), it's always a fail after 7 hours.

Any ideas? It's just... an error after 7 hours for a try per day... And the log does not help me understand where could be the issue.

Thanks for any advices.

Regards,
 
It was built fine for me on 143amd64.
Code: 
% tail /poudriere/data/logs/bulk/143amd64-default/latest-per-pkg/rust-1.90.0_1.log
===========================================================================
=======================<phase: package        >============================
===== env: 'PKG_NOTES=build_timestamp ports_top_git_hash ports_top_checkout_unclean port_git_hash port_checkout_unclean built_by' 'PKG_NOTE_build_timestamp=2025-10-29T21:06:05+0000' 'PKG_NOTE_ports_top_git_hash=5195754ac' 'PKG_NOTE_ports_top_checkout_unclean=no' 'PKG_NOTE_port_git_hash=d1ec81a0d' 'PKG_NOTE_port_checkout_unclean=no' 'PKG_NOTE_built_by=poudriere-git-3.4.4' NO_DEPENDS=yes USER=root UID=0 GID=0
===>  Building packages for rust-1.90.0_1
===>   Building rust-1.90.0_1
===========================================================================
=>> Cleaning up wrkdir
===>  Cleaning for rust-1.90.0_1
build of lang/rust | rust-1.90.0_1 ended at Thu Oct 30 01:52:44 +03 2025
build time: 01:46:38
 
Hi.

Here some answers.

dmesg -a
Code: 
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4206391, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4206391, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4206391, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4209537, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4228229, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4211052, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4209537, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4228229, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4989044, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4989044, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4989044, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4989044, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978286, size: 12288
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978234, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978243, size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978309, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978290, size: 28672
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978305, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978297, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978300, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978305, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978297, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978300, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978305, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978297, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978300, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978305, size: 16384
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978300, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4572814, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4978305, size: 16384
pid 54576 (rust-installer), jid 12, uid 65534, was killed: failed to reclaim memory

So, a memory issue? It's strange, because at the moment of the crash, my swap was not full. At the moment of the crash (after 7 hours), swap is used at 30%.
One month ago, I had swap issue (my 16 Go was not enough), so I add a second 16 Go swap file, and everything was fine. It's just since the last week issues are back.

swapinfo -h
Code: 
Device              Size     Used    Avail Capacity
/dev/ada0p2          16G       0B      16G     0%
/dev/md0             16G       0B      16G     0%
Total                32G       0B      32G     0%

Another fail this night:
Code: 
Copy/Link "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/LICENSE.TXT" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/LICENSE.TXT"
Install "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/README.txt" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/README.txt"
Copy/Link "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/llvm-project/llvm/README.txt" to "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust-dev/x86_64-unknown-freebsd/overlay/README.txt"
  c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false } }
  c RustInstaller { compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false }, target: x86_64-unknown-freebsd }
  c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-unknown-freebsd, forced_compiler: false } }
Dist rust-dev-1.90.0-x86_64-unknown-freebsd
Command has failed. Rerun with -v to see more details.
Traceback (most recent call last):
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/x.py", line 53, in <module>
    bootstrap.main()
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 1377, in main
    bootstrap(args)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 1352, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/wrkdirs/usr/ports/lang/rust/work/rustc-1.90.0-src/src/bootstrap/bootstrap.py", line 237, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /wrkdirs/usr/ports/lang/rust/work/_build/bootstrap/debug/bootstrap dist --jobs=1
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/rust
=>> Cleaning up wrkdir
===>  Cleaning for rust-1.90.0_1
build of lang/rust | rust-1.90.0_1 ended at Tue Nov 11 05:22:22 CET 2025
build time: 07:00:54
!!! build failure encountered !!!
 
I'm swapping on /dev/ada0p2 (16 Go). Since a month, it was not enough (for building lang/rust), so, I added a file swap (with dd, dd and swapon. And to be honest, everything was OK until last week. I know it's dirty, but it's only for lang/rust (I don't even use it, it's only a build dependencies for other ports).

I can't increase /dev/ada0p2, and I didn't want touching or resetting slices of the hdd.
I can't add another hdd.

gpart show
Code: 
=>       40  500118112  ada0  GPT  (238G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   33554432     2  freebsd-swap  (16G)
   33556480  466561024     3  freebsd-zfs  (222G)
  500117504        648        - free -  (324K)

Probably I have two solutions:
- add more RAM (and I don't know if the machines will allow it)
- set a properly swap slice (32 Go ?)

And what about swapping on a single 32 Go md device ? Or a single swap file (with mkswap)?
Thanks for your answers, and ideas. I'll read every advices.
 
You might get away with using an USB stick for this. File handling needs memory itself, so swapping to files can run into out of memory itself while trying to swap out memory to free memory. I had a swap file for some time, and I ran into trouble with it.
 
Ah, thx for the tip. However, will performances be OK? I mean: my hard disk in a SSD.
And, do I need to remove my two swaps (/dev/ada0p2 and the md swap file) in advantage of a USB stick?
 
I would suggest to remove the swap file. Also, swap does not need to be that fast. Idle memory is swapped out, and in later. The file system usually sees more traffic. Heck, back in the days, we had systems which had swap over network. And it would get you out of that pit you are currently in. Also, how fast is it building right now? Not at all - so everything will be a speed up.
 
I'll try tonight.
But, to be more precise, you suggest:
- a combo swap with /dev/ada0p2 and a USB stick?
- or just the USB stick?
 
It is probably overall faster to reduce parallelism during the Rust build than running it with heavy paging. Depends on how much time is spent in the overflowing periods of course.

I am unsure how to control the parallelism on the Rust build, though.
 
You can check in top(1) to see whether parallel jobs are running.

OP,, how much RAM do you have? I would fine it strange to OOM with 1 parallel build job.
 
cracauer@ said:
OP,, how much RAM do you have? I would fine it strange to OOM with 1 parallel build job.
Click to expand...
It seens this is not a case of OOM, but if it is, and it is only one job, that sheds a flickering light on rust and the ecosystem around it...
 
The machine has 16 Go of RAM. Build uses default values, so I guess it does not use parallelism?
More, build of lang/rust is OK (in 07:02:50) with a swap based on /dev/ada0p2 and a USB stick, so thanks!
Probably issue was using a md swap?

After the build:
swapinfo -h

Code: 
Device              Size     Used    Avail Capacity
/dev/ada0p2          16G     382M      16G     2%
/dev/da1p1           14G     382M      14G     3%
Total                30G     764M      30G     2%
 
ssbear said:
Probably issue was using a md swap?
Click to expand...
The question is how that was set up - was it file based or not? Because if you made a mistake there and it was not, then it is memory based - and swapping memory to memory when memory gets low ... does not work out.
 
Looks good, but then -
ssbear said:
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4205958, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4210375, size: 24576 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4206391, size: 12288
Click to expand...
Looks like the file system can not find the memory to page in something. It is best to avoid this complexity. If you are using ZFS, a vdev can be better used. And maybe you need to adjust the minfree sysctrl, so there is always some memory free to do something.
 
Yes, I’m using ZFS.
I deleted the md swap for a usb stick, waiting a better solution: probably I’ll buy more RAM (from 2x8 Go to 2x16 Go) or set a better slice swap (from 16 Go to 32 Go) and do a proper re-install with FreeBSD 15.0-RELEASE in december.

Thanks a lot for your advices and answers!
 
Honestly, I don’t know, I use default options.
But I’ll take every advices to optimize the build of lang/rust for small servers. In my case, it has to be re-built every weeks (for an update of the port or a build dependency), and it takes 7 hours.
 
On my system with 8 cores + hyperthreading for 16 hardware threads Rust compiles with 14 parallel processes most of the time. The processes are staying within 1 GB each. Some processes multithread to about 500% CPU time.

What CPU do you have?
 
Code: 
CPU: Intel(R) Core(TM) i5-6500T CPU @ 2.50GHz (2500.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3

So, 4 threads
 
rust has always been a major PITA to build due to its idiotically high ressource usage (which is considered 'normal' and 'ok' by its devs...), which is also constantly increasing. Currently it piles up ~60GB in tmpfs during build. So, first and foremost you have to add rust to the TMPFS_BLACKLIST in poudriere.conf to prevent it from exhausting all memory during build. Without this I had rust failing (and often causing other fallout due to complete memory exhaustion) even on hosts with 256GB or more of RAM...
 
You must log in or register to reply here.
Back
Top