Building the entire pkg repo with Poudriere

V

vmb

I want to build the entire pkg repo using Poudriere. Is there an easy way to get the entire list of packages with their origin?

pkg search .

Gives the entire list with version numbers and descriptions but it doesn't give the origin.

E.g.
nginx-1.28.2_11,3 Robust and small WWW server

With origin:

www/nginx
 
All packages like 'make package' for te entire portstree? List all directories under /usr/ports/<category> and pipe it through a make BATCH=1 command.
I don't think anybody has that space at home. Maybe if you delete the work dir and source file immediately after all makes. 😎
 
It's best to stick to one python version.
For me :
- DEFAULT_VERSIONS+=python=3.11

I tried once to install all python packages of this version, even that was a failure.
All llvms no problem.
 
  • Thanks
Reactions: vmb
Before I start this build process, does anyone have any idea how much disk space will be required on my Poudriere host for just the amd64 pkg build for FreeBSD 15.0-RELEASE ? I will only be building packages for that one target. The storage for the finished packages will be on a separate web server, Poudriere will mount it with NFS.

What CPU will I need to complete a full build cycle every three months? I am planning to use an i7-4785T 35W TDP, 2.2Ghz, 4-core, 8-thread. It runs 8 build jails simultaneously just fine.

An alternative, will be to just to take the sources for the entire ports tree and use Poudriere to build what I am currently using. I would like to be prepared for when the country I live in starts sanitizing the internet for citizens by blocking sites the government doesn't like. I am taking the view that this could affect my access to source code and package repositories. I can live without the internet if I have source code from a snapshot in time. Building packages on Poudriere is a good way of proving what can be built successfully once all of the distfiles have been obtained from the remote sources. If I just take a copy of the source code for the ports tree, I wont actually know what builds and what doesn't until the time when I want to use that code. By then it could be too late to do anything about it.

Back in the 1980's before the publicly accessible internet was a thing, I managed to do all the things I wanted to do with an open source and shareware software library that I had put together by legally purchasing copies distributed on floppy disks. I found a CD-ROM carry case that I used to take to work 30 years ago that had inside it CDRs that I had made of my library. I have no idea if they are still useable, but it reminded me how easy and normal it was not having access to the internet or CompuServe when I had everything that I needed on portable media.

I guess I am prepping digitally for a heavily censored and sanctioned internet. As I can still remember automating unattended builds of Windows 3.1 for corporates that had no internet access, losing the internet isn't scary for me. Documentation is the bigger problem. With so much provided now on JavaScript heavy websites, saving documentation as PDFs is a chore if it is not available natively.

Yesterday, I resurrected my custom kernel build machine and it successfully completed a buildworld and buildkernel for freebsd16. I haven't built a custom kernel since FreeBSD 12. I think I may move this to my Poudriere host and start looking at pkg base, building and running the custom kernel on my Poudriere host before package building.
 
Configuring poudriere.conf & make.conf can be a process of trial and error.
It depends on how many cpu cores you have, how much swap , disk speed.
Best is to store everything on NVME disk with ZFS filesystem.

I currently rebuilding 1800 packages,
This is my config
cat /root/Scripts/poudriere.conf
Code: 
ZPOOL=SSD
ZROOTFS=/usr/local/poudriere
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS="all"
TMPFS_LIMIT=10
DISTFILES_CACHE=/usr/ports/distfiles
BUILDS=2
BUILDERS=2
PARALLEL_JOBS=2
PREPARE_PARALLEL_JOBS=3
MAKE_JOBS_NUMBER=10
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="llvm* gcc* rust* waterfox firefox*"
TIMESTAMP_LOGS=yes
BUILDER_HOSTNAME=myfreebsd
USE_COLORS=yes
URL_BASE=http://127.0.0.1/poudriere/
PACKAGE_FETCH_BRANCH=quarterly

pkg-base is needed for nothing ...

As i don't use "experimental stuff" as host i'm on FreeBSD-15.0-RELEASE-p1.
For poudriere jaul , "poudriere jail -c -j $myjail -v 15.0-RELEASE"

In my case i monitored, cpu-usage & swap-usage.
When 15% swap usage is GOOD. When lower increase "BUILDS,BUILDERS,PARALLEL_JOBS,PREPARE_PARALLEL_JOBS,MAKE_JOBS_NUMBER ." ; when higher lower them.
I''ve seen CPU was not used alot in my case thats why i put,
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="llvm* gcc* rust* waterfox firefox*"
So everything is done in memory , and cpu usage increases , you can check load with top.
For a 12-core CPU , system with CPU-load 9 , you have a responsive desktop.
Success !
 
  • Thanks
Reactions: vmb
vmb said:
What CPU will I need to complete a full build cycle every three months? I am planning to use an i7-4785T 35W TDP, 2.2Ghz, 4-core, 8-thread. It runs 8 build jails simultaneously just fine.
Click to expand...

Errr. That's not gonna work.

The problem is that it will take way too much time to go from one build failure to the next.
 
cracauer@ said:
Errr. That's not gonna work.

The problem is that it will take way too much time to go from one build failure to the next.
Click to expand...
I am familiar with Poudriere. I want to initiate a build cycle every quarter while I still have internet access. I am expecting build problems that will need manual intervention. I will check and make corrections daily until the entire process completes. It wont be a case that I will check on it every 90 days.

I can use a better processor like a Ryzen 5600X that would give me 12-cores. The 5600X TDP is nearly double that of the i7-4785T, but it has roughly four times the performance on threads. If the Ryzen completes in at least half the time, I haven't spent any more on electricity. The machine will only be used for kernel compilation and Poudriere.

I have noticed that some packages need more than 8GB to build on Poudriere so the machine will likely have 16GB to start with. A four DIMM slot motherboard would be sensible for future expansion.

I hadn't really thought about it before, but I guess having 8GB-16GB of swap on NVME would be beneficial as swap performance would be quick if Poudriere got to a point where it was building multiple heavyweight packages simultaneously (llvm etc). The machine wouldn't need mirrored drives as I would use ZREPL to fanout source code/distfiles dataset replicas to my RAIDZ2 backup servers. When the time comes that I lose access to the internet, that machine will never have to build packages again until I need to modify source code or pkg options.
 
I'm running poudriere on my daily-driver PC with Core i9-12900H (6PC8EC20T), 64GiB of RAM and RTX A400 GPU limiting 6 builder jail at the same time, allowing
  • TMPFS [blocklisting chromium llvm* webkit* *webengine*], WITH_CCACHE_BUILD,
  • ALLOW_MAKE_JOBS
  • ALLOW_MAKE_JOBS_PACKAGES [pkg ccache py* rust* llvm* gcc* firefox* chromium* webkit* libreoffice* stellarium* *webengine* linux-ai-ml-env*]
and STILL SWAPS, around 32GB at maximum that I could see.
 
Alain De Vos said:
The number of cores, are not that important. One just has to wait abit longer. Memory is key.
Click to expand...
I thought 64GiB should be sufficient as my previous daily-driver, ThinkPad P52 with Core i7-8750H with 6C12T, 32GiB RAM and Quadro P1000, swapped around 30GiB at maximum with same /usr/local/poudriere.conf (only pool name differs) and 6 builder jails in parrarrel (the same).

Maybe the number of cores matters how much RAMs are consumed.
 
You must log in or register to reply here.
Back
Top