Go is much smaller and lighter than Rust itself though...And building go instead of rust and replacing one horribly bloated buildjob with another...
Nope, I don't use it on this machine (only 8 Go RAM). I'll try with one job, and if it fails again, I'll buy an older workstation (with a i5 CPU / 16 GO RAM) and set a poudriere on it, at my home. It will be less expensive than using ec2 instances and ressources.Yeah, -j 1 is the next thing to try. Do you use ZFS there?
?We should treat building Rust as a sport.
PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_WHITELIST="rust"
PACKAGE_FETCH_URL=pkg+http://pkg.FreeBSD.org/\${ABI}
pkg info | grep rust returns:rust-1.71.0 Language with a focus on memory safety and concurrency
poudriere bulk -j 132Ramd64 security/py-certbot, it still trying to build devel/rust...DEFAULT_VERSIONS+=pycryptography=legacy
I'll buy an older workstation (with a i5 CPU / 16 GO RAM) and set a poudriere on it, at my home.
My old i5 (3470; 2 cores/4 threads; 16 GB) runs ZFS and poudriere. It's my "build" server. Yeah, it's not the fastest in the world, but it works just fine.Not sure it will work: on my build server 16 GB RAM are not enough.
for j in ${SERVER_REPOS}; do
${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/rust.lst
${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/gcc.lst
${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/llvm.lst
${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/${j}-server-package.lst
done
My old i5 (3470; 2 cores/4 threads; 16 GB) runs ZFS and poudriere. It's my "build" server. Yeah, it's not the fastest in the world, but it works just fine.
Yes, default setting of Poudriere.Do you use TMPFS?
USE_TMPFS=yes
tmpfs on /tmp (tmpfs, local)
Sure it can help (and I still think it's the most convenient solution if you don't fiddle with anything that would affect the rust package). It just requires these extra steps making sure your ports tree is on the exact commit from which the package repo was built to work reliably....I understand, thanks for your answer.
So, ports-mgmt/poudriere-devel does not help me.
Here's the script you posted in another thread to get the port hash - it does work but I haven't incorporated it into my build script to use Poudriere's Fetch feature:Sure it can help (and I still think it's the most convenient solution if you don't fiddle with anything that would affect the rust package). It just requires these extra steps making sure your ports tree is on the exact commit from which the package repo was built to work reliably....
# List of package globs that are not allowed to use tmpfs for their WRKDIR
# Note that you *must* set TMPFS_BLACKLIST_TMPDIR
# EXAMPLE: TMPFS_BLACKLIST="rust"
update-ports command is the one that does what zirias is describing:#!/bin/sh
commit=$(fetch -q -o - 'https://pkg-status.freebsd.org/api/1/builds?type=package&jailname=132amd64' | fx 'x["builds"].find(b => b.mastername == "132amd64-default" && b.status == "stopped:done:").buildname')
if [ -z $commit ]; then
echo 'Could not find completed build. Still running? https://pkg-status.freebsd.org/' 1>&2
exit 1
fi
cd freebsd-ports.git/main && git remote update && git reset --hard $commit
-b latest (as used in the build-prod command) to fetch packages.-c to delete previously downloaded packages, because options have changed between the fetched version and git, and poudriere thinks it needs to rebuild.-n -v to see what triggers a build for a package that you expect to be downloaded.As weird as it sounds, in such a case it works even better to just interrupt poudriere after it deleted packages not matching any more and immediately restart it ... then it will download just these packages ?Every once in a while I have to build with-cto delete previously downloaded packages, because options have changed between the fetched version and git, and poudriere thinks it needs to rebuild.
Another try with a better ec2 instance, with 16 Go RAM, 8 Go swap (no ZFS, /tmp as tmpfs, with one job) : stil a fail after 5 hours.Not sure it will work: on my build server 16 GB RAM are not enough. But I use ZFS, maybe with UFS will work for you.
Another try with a better ec2 instance, with 16 Go RAM, 8 Go swap (no ZFS, /tmp as tmpfs, with one job) : stil a fail after 5 hours.
Even on my 64GiB server I use to build my repository, I still use poudriere's package pre-fetching, because there's just no point in building exactly the same thing locally that a FreeBSD build server already did.
I understand your point. What I was trying to say is, for me, FreeBSD has always been a great OS for light and old machines. I use it since 2006, with... 256 Mo RAM (I could even build src with this). Of course, I had to increase the RAM and CPU years after years, yes. But increase from 4 Go to 64 Go is a huge gap... for a single port.Two things to remark here:
1. It's not FreeBSD deciding which languages third-party projects use, and just sticking to an outdated version (especially in a security context) is certainly not an option.
2. Compilers with tooling and "standard libs" these days are massive projects, there's no way to change that either. One of them, llvm (including what's necessary for C and C++) is part of base and indeed takes the vast majority of resources when building base from source. Rust is not in base, so, must be built from a port. If you install base from binary releases, but build ports yourself, you're getting a somewhat wrong picture here.
For massive builds these days, 16GiB of RAM is still kind of limited. I'd really still recommend to only build yourself what you must (because you need non-default build options) and use as many binary packages as possible. Even on my 64GiB server I use to build my repository, I still use poudriere's package pre-fetching, because there's just no point in building exactly the same thing locally that a FreeBSD build server already did.
DEFAULT_VERSIONS+=pycryptography=legacy