Rust building failed

ssbear · Sep 15, 2023

Just an update to explain that using

Code:

DEFAULT_VERSIONS+=pycryptography=legacy

is not a good solution if you use security/py-certbot or devel/awscli
Impact, when using them:

Code:

Traceback (most recent call last):
 File "/usr/local/bin/aws", line 19, in <module>
   import awscli.clidriver
 File "/usr/local/lib/python3.11/site-packages/awscli/clidriver.py", line 17, in <module>
   import botocore.session
 File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
   import botocore.client
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
   from botocore import waiter, xform_name
 File "/usr/local/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
   from botocore.docs.docstring import WaiterDocstring
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
   from botocore.docs.service import ServiceDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
   from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
   from botocore.docs.example import ResponseExampleDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
   from botocore.docs.shape import ShapeDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
   from botocore.utils import is_json_value_header
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
   import botocore.httpsession
 File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
   from urllib3.contrib.pyopenssl import (
 File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
   import OpenSSL.crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'
Traceback (most recent call last):
 File "/usr/local/bin/aws", line 19, in <module>
   import awscli.clidriver
 File "/usr/local/lib/python3.11/site-packages/awscli/clidriver.py", line 17, in <module>
   import botocore.session
 File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
   import botocore.client
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
   from botocore import waiter, xform_name
 File "/usr/local/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
   from botocore.docs.docstring import WaiterDocstring
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
   from botocore.docs.service import ServiceDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
   from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
   from botocore.docs.example import ResponseExampleDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
   from botocore.docs.shape import ShapeDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
   from botocore.utils import is_json_value_header
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
   import botocore.httpsession
 File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
   from urllib3.contrib.pyopenssl import (
 File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
   import OpenSSL.crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'
Traceback (most recent call last):
 File "/usr/local/bin/aws", line 19, in <module>
   import awscli.clidriver
 File "/usr/local/lib/python3.11/site-packages/awscli/clidriver.py", line 17, in <module>
   import botocore.session
 File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
   import botocore.client
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
   from botocore import waiter, xform_name
 File "/usr/local/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
   from botocore.docs.docstring import WaiterDocstring
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
   from botocore.docs.service import ServiceDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
   from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
   from botocore.docs.example import ResponseExampleDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
   from botocore.docs.shape import ShapeDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
   from botocore.utils import is_json_value_header
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
   import botocore.httpsession
 File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
   from urllib3.contrib.pyopenssl import (
 File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
   import OpenSSL.crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'
Traceback (most recent call last):
 File "/usr/local/bin/aws", line 19, in <module>
   import awscli.clidriver
 File "/usr/local/lib/python3.11/site-packages/awscli/clidriver.py", line 17, in <module>
   import botocore.session
 File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
   import botocore.client
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
   from botocore import waiter, xform_name
 File "/usr/local/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
   from botocore.docs.docstring import WaiterDocstring
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
   from botocore.docs.service import ServiceDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
   from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
   from botocore.docs.example import ResponseExampleDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
   from botocore.docs.shape import ShapeDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
   from botocore.utils import is_json_value_header
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
   import botocore.httpsession
 File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
   from urllib3.contrib.pyopenssl import (
 File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
   import OpenSSL.crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'
Traceback (most recent call last):
 File "/usr/local/bin/aws", line 19, in <module>
   import awscli.clidriver
 File "/usr/local/lib/python3.11/site-packages/awscli/clidriver.py", line 17, in <module>
   import botocore.session
 File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 26, in <module>
   import botocore.client
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 15, in <module>
   from botocore import waiter, xform_name
 File "/usr/local/lib/python3.11/site-packages/botocore/waiter.py", line 18, in <module>
   from botocore.docs.docstring import WaiterDocstring
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/__init__.py", line 15, in <module>
   from botocore.docs.service import ServiceDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in <module>
   from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/client.py", line 17, in <module>
   from botocore.docs.example import ResponseExampleDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in <module>
   from botocore.docs.shape import ShapeDocumenter
 File "/usr/local/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in <module>
   from botocore.utils import is_json_value_header
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 37, in <module>
   import botocore.httpsession
 File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
   from urllib3.contrib.pyopenssl import (
 File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
   import OpenSSL.crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'

Code:

Traceback (most recent call last):
 File "/usr/local/bin/certbot", line 33, in <module>
   sys.exit(load_entry_point('certbot==2.6.0', 'console_scripts', 'certbot')())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/bin/certbot", line 25, in importlib_load_entry_point
   return next(matches).load()
          ^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
   module = import_module(match.group('module'))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
   return _bootstrap._gcd_import(name[level:], package, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
 File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
 File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 940, in exec_module
 File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 File "/usr/local/lib/python3.11/site-packages/certbot/main.py", line 6, in <module>
   from certbot._internal import main as internal_main
 File "/usr/local/lib/python3.11/site-packages/certbot/_internal/main.py", line 21, in <module>
   import josepy as jose
 File "/usr/local/lib/python3.11/site-packages/josepy/__init__.py", line 40, in <module>
   from josepy.json_util import (
 File "/usr/local/lib/python3.11/site-packages/josepy/json_util.py", line 14, in <module>
   from OpenSSL import crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
   from OpenSSL import SSL, crypto
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/SSL.py", line 19, in <module>
   from OpenSSL.crypto import (
 File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 3263, in <module>
   utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'

Everything's ok when using old built packages py39-certbot and py39-awscli.
Really tired of this issue (I spent a week on it...), so everything is built from ports, except py39-certbot and py39-awscli (built packages) which cohabitate with python311 (ports)... Dirty, but no other solution...

zirias@ · Sep 15, 2023

I guess I lost track here, did you try poudriere-devel's package pre-fetching option, e.g. with the script posted by patmaddox (which looks pretty sane and simple)? I'd still argue this should be the "best" solution....

ssbear · Sep 15, 2023

I will definitely try in the next weeks. I will order a new local machine, and set up a poudriere server on it at home. This script will certainly be very useful!

ssbear · Oct 7, 2023

End of the story. I’ve bought an old Lenovo ThinkCenter, just to have a Poudriere machine.
Intel core i5 / 16 Go RAM / ZFS on a ssd drive / 16 Go swap

Build of lang/rust worked! (5 hours).

In my case, it’s a better and simpler solution than pay ec2 time and ressources!

SirDice · Oct 11, 2023

ssbear said:
Build of lang/rust worked! (5 hours).

Good, it worked. Takes a lot longer though, on my old i5 it takes a little over 2 hours. Did you add rust to ALLOW_MAKE_JOBS_PACKAGES in poudriere.conf?

amendlik · Oct 31, 2023

I've been following this thread and trying to get lang/rust to build on my own Poudriere server. So far I've:

Increased the RAM on my server to 16GiB
Added a 32GiB swap device
Added ALLOW_MAKE_JOBS_PACKAGES to my poudriere.conf file

Now, the build makes it to the end in about an hour, but still "crashes" and leaves Poudriere hung. Here are the last few lines of the build log:

Code:

Dist rust-1.73.0-x86_64-unknown-freebsd
running: LD_LIBRARY_PATH="/wrkdirs/usr/ports/lang/rust/work/_build/x86_64-unknown-freebsd/stage0-bootstrap-tools/x86_64-unknown-freebsd/release/deps:/wrkdirs/usr/ports/lang/rust/work/bootstrap/lib" RUSTC="/wrkdirs/usr/ports/lang/rust/work/bootstrap/bin/rustc" "/wrkdirs/usr/ports/lang/rust/work/_build/x86_64-unknown-freebsd/stage0-tools-bin/rust-installer" "combine" "--input-tarballs" "/wrkdirs/usr/ports/lang/rust/work/_build/dist/rustc-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rust-std-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rust-docs-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rust-docs-json-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/cargo-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rustfmt-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rls-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rust-analyzer-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/clippy-1.73.0-x86_64-unknown-freebsd.tar.gz,/wrkdirs/usr/ports/lang/rust/work/_build/dist/rust-analysis-1.73.0-x86_64-unknown-freebsd.tar.gz" "--rel-manifest-dir=rustlib" "--legacy-manifest-dirs=rustlib,cargo" "--product-name=Rust" "--success-message=rust installed." "--package-name=rust-1.73.0-x86_64-unknown-freebsd" "--non-installed-overlay" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust/x86_64-unknown-freebsd/overlay" "--output-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/dist" "--work-dir" "/wrkdirs/usr/ports/lang/rust/work/_build/tmp/tarball/rust/x86_64-unknown-freebsd" "--compression-profile" "fast"
    finished in 20.384 seconds
< Extended { stage: 2, host: x86_64-unknown-freebsd, target: x86_64-unknown-freebsd }
> ReproducibleArtifacts { target: x86_64-unknown-freebsd }
< ReproducibleArtifacts { target: x86_64-unknown-freebsd }
Build completed successfully in 1:04:16
===========================================================================
Build crashed: run-depends

Does anyone know why my build might be completing successfully AND crashing at the same time?

cracauer@ · Oct 31, 2023

Why does it say "run-depends" crashed when it obviously just built rust, not a dependency?

PMc · Nov 1, 2023

I never perceived a memory issue with rust. My build instances get configured with
3711 MB + 1237 MB/core
That is less than 8 GB for 2 cores (and uses ZFS).

rust build ressource consumption: doesn't swap

Code:

ncpu 2
real 24522.51
user 45063.92
sys 1654.98
   1364800  maximum resident set size
     34377  average shared memory size
       257  average unshared data size
       194  average unshared stack size
 146953744  page reclaims
    150363  page faults
         0  swaps
    128099  block input operations
   1215586  block output operations
     80449  messages sent
    193954  messages received
      2169  signals received
   1803922  voluntary context switches
  25460749  involuntary context switches

In comparison this is llvm15, and that one does swap (6 cores):

Code:

ncpu 6
real 17098.39
user 83570.47
sys 4165.07
   6025020  maximum resident set size
     60134  average shared memory size
       450  average unshared data size
       363  average unshared stack size
 316485685  page reclaims
    908570  page faults
        18  swaps
     45386  block input operations
    410706  block output operations
     16059  messages sent
     34319  messages received
      1778  signals received
   1960086  voluntary context switches
   4955405  involuntary context switches

cracauer@ · Nov 1, 2023

A LLVM build is temporarily very large because it runs very big linker processes. I don't think Rust has such spikes, but it might be bigger on average.

zirias@ · Nov 1, 2023

cracauer@ said:
Why does it say "run-depends" crashed when it obviously just built rust, not a dependency?

run-depends is always executed after build (obviously these are dependencies not required for building). By default, the ports framework would build these recursively, but poudriere adds USE_PACKAGE_DEPENDS_ONLY=1, so run-depends would only install already existing packages. Nevertheless, it is executed. This makes sense, it would catch issues with e.g. installation conflicts, and it's even possible some run dependency could be required for successful staging.

Of course this doesn't explain why/how it's crashing. Just a wild guess, this might be another "running out of memory" issue with tmpfs(5) usage. There's a reason the example comments in /usr/local/etc/poudriere.conf have this:

Code:

# List of package globs that are not allowed to use tmpfs for their WRKDIR
# Note that you *must* set TMPFS_BLACKLIST_TMPDIR
# EXAMPLE: TMPFS_BLACKLIST="rust"

amendlik · Nov 7, 2023

I've added TMPFS_BLACKLIST="rust" to my poudriere.conf but it had no effect. I'm still getting the same error.

T-Aoki · Nov 7, 2023

zirias@ said:
run-depends is always executed after build (obviously these are dependencies not required for building). By default, the ports framework would build these recursively, but poudriere adds USE_PACKAGE_DEPENDS_ONLY=1, so run-depends would only install already existing packages. Nevertheless, it is executed. This makes sense, it would catch issues with e.g. installation conflicts, and it's even possible some run dependency could be required for successful staging.

Ah, thanks! This made sense for me. Sometimes build fails on staging phase in poudriere[-devel] jail.
Not sure it's always true, but at least for some cases, I've noticed that poudriere builds RUN_DEPENDS at the last phase of builder. This would be the reason.
And if the absense of any port(s) listed on RUN_DEPENDS cause(s) breakage on staging, it (they) SHALL be listed as BUILD_DEPENDS, too.
This seems to be one of the reason which forces multiple run of poudriere[-devel] to finish.

amendlik · Nov 13, 2023

Any advice on how I should diagnose this error?

spectrum48 · Nov 13, 2023

amendlik said:
Any advice on how I should diagnose this error?

I'm not 100% sure but I think I had to install poudriere-devel instead of poudriere because this is a relatively new feature.

abqcheeks · Dec 18, 2023

I had this same problem, on a 16GB machine. Tried several things, and going to poudreire-devel and setting TMPFS_BLACKLIST is what finally worked.

I think part the discrepancy in build times people are reporting might be down to some using CCACHE and some not. My first (failed) build took 11.5 hours. Subsequent failures took 60-90 minutes. The successful build with TMPFS_BLACKLIST took 78 minutes.

Rust building failed

Administrator