Poudriere, PACKAGE_FETCH_WHITELIST, and how do I avoid building rust?

P

putney

I've got a 14.0-RC1 VM running on a Macbook Air, which I'm using to try to get to understand poudriere a little.

My VM doesn't have the resources to build rust, so I'm using poudriere-devel. In /usr/local/etc/poudriere.conf I have added

Code: 
#PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_URL=pkg+http://pkg.FreeBSD.org/\${ABI}
PACKAGE_FETCH_BLACKLIST=""
PACKAGE_FETCH_WHITELIST="gcc* rust llvm*"

& I've make a poudriere jail, poo14.

poudriere still wants to build rust, saying

Code: 
root@fair14:~ # cat /usr/local/etc/poudriere.d/poo14-make.conf 
lang_rust_UNSET=PORT_LLVM
root@fair14:~ # poudriere bulk -j poo14 -p default -f /usr/local/etc/poudriere.d/packages-default -b latest -v
...
[00:00:08] Package fetch: Skipping rust-1.72.0: options wanted: +DOCS -GDB -LTO -PORT_LLVM +SOURCES +WASM
[00:00:08] Package fetch: Skipping rust-1.72.0: options remote: +DOCS -GDB -LTO +SOURCES +WASM
[00:00:08] Package fetch: No packages eligible to fetch

I'm sure there's more than one thing I've misunderstood, but how do I get past this thing wanting PORT_LLVM?
 
You have different options set. So you can't use the package version from the FreeBSD repository. Remove this lang_rust_UNSET=PORT_LLVM.

Code: 
              poudriere will only use packages that:
              •   come from a repository having the same or older version of
                  pkg.
              •   do not have a locally fetched package already.
              •   are not IGNORED.
              •   match the expected local version.
              •   match the expected ABI.
              •   match the expected runtime and library dependencies.
              •   match the expected OPTIONS when CHECK_CHANGED_OPTIONS is
                  enabled (default: on).
              •   is NOT listed in PACKAGE_FETCH_BLACKLIST.
              •   is NOT listed with -C, or -c, when -t is used.
 
SirDice said:
You have different options set. So you can't use the package version from the FreeBSD repository. Remove this lang_rust_UNSET=PORT_LLVM.
Click to expand...

Thanks. I tried that in /usr/local/etc/poudriere.d/poo14-make.conf, as above - is this the wrong place for it?
 
You should remove those options, regardless in which make.conf you added them. The options are different from the options in the FreeBSD repositories and poudriere therefor will not use that package. In order to use the FreeBSD package the options MUST be exactly the same, in other words, they have to be the default (packages in the FreeBSD repositories are always built with their default options).
 
I don't have /etc/make.conf. My /usr/local/etc/poudriere.d/poo14-make.conf only had lang_rust_UNSET=PORT_LLVM as you suggested (and I've got rid of that now). I never heard of PORT_LLVM until I found poudriere insisting on compiling rust; I certainly have not deliberately set it.

Rust's makefile doesn't set this option as I understand it:
Code: 
root@fair14:/usr/local/etc/poudriere.d # grep OPTIONS /usr/ports/lang/rust/Makefile
OPTIONS_DEFINE=        DOCS GDB LTO PORT_LLVM SOURCES WASM
OPTIONS_DEFAULT=    SOURCES WASM
.if ${PORT_OPTIONS:MWASM} && ${PORT_OPTIONS:MDOCS}
.if ${PORT_OPTIONS:MWASM} && !${PORT_OPTIONS:MPORT_LLVM}
.if ${PORT_OPTIONS:MLTO}
.if ${PORT_OPTIONS:MLTO}
.if ${PORT_OPTIONS:MPORT_LLVM}
root@fair14:/usr/local/etc/poudriere.d #

I'm at a loss. "regardless in which make.conf" - are there others? I'll readily admit to being confused, and I may have made mistakes while blundering with poudriere or portconfig.
 
putney said:
I'm at a loss. "regardless in which make.conf" - are there others?
Click to expand...
Poudriere ignores the system's /etc/make.conf, so that one doesn't matter. Poudriere can have a bunch of make.conf files in /usr/local/etc/poudriere.d.
Code: 
   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf
This allows for some interesting combinations.
Code: 
root@molly:/usr/local/etc/poudriere.d # ll *make.conf
-rw-r--r--  1 root wheel 701 Aug 27 17:04 desktop-make.conf
-rw-r--r--  1 root wheel 336 Jun 17 14:47 gitlab-make.conf
-rw-r--r--  1 root wheel 588 Aug  2 11:49 make.conf
-rw-r--r--  1 root wheel 632 Aug 20 23:08 server-make.conf
I create several different repositories. I have a 'desktop' and a 'server' set for example because I different requirements/needs for a server or desktop configuration. This then gets combined with different version FreeBSD jails.
Code: 
root@molly:/usr/local/etc/poudriere.d # poudriere jail -l
JAILNAME    VERSION         ARCH  METHOD                                    TIMESTAMP           PATH
13-stable   13.2-STABLE     amd64 url=file:///storage/release/13-stable/ftp 2023-08-13 22:01:59 /usr/local/poudriere/jails/13-stable
132-release 13.2-RELEASE-p3 amd64 http                                      2023-09-30 16:48:22 /usr/local/poudriere/jails/132-release
14-stable   14.0-STABLE     amd64 url=file:///storage/release/14-stable/ftp 2023-09-30 16:44:25 /usr/local/poudriere/jails/14-stable
140-release 14.0-BETA3      amd64 http                                      2023-09-30 16:45:59 /usr/local/poudriere/jails/140-release

For a 'server' build make.conf and server-make.conf get combined. For a 'desktop' build it's make.conf and desktop-make.conf. This means I can set some common defaults in make.conf and 'server' specific options go in server-make.conf.
 
I think I've got a basic understanding of what was going wrong now. I hope the following is correct :)

lang/rust in ports has, for a while, had a PORT_LLVM option in OPT_DEFINED:
Code: 
root@fair14:/usr/local/poudriere/ports/default/lang/rust # grep OPT Makefile 
OPTIONS_DEFINE=        DOCS GDB LTO PORT_LLVM SOURCES WASM
..

But the prebuilt packages available (at least for aarch64) didn't have this option. This seems to have been a temporary situation - see <https://cgit.freebsd.org/ports/commit/?id=098de5bc2195a064523eb32e7db6071b2f8ed93a>.

Poudriere distinguishes between (1) not having the option and (2) having the option, but it's being turned off. So it wanted to build rust locally.

putting
Code: 
lang_rust_UNSET=PORT_LLVM
in any make.conf turns it off. It doesn't make it as though the option never existed, so this doesn't make it possible to use a fetched package that doesn't know about the option.

The situation now is that I _still_ have to build rust locally because the port sources want v1.73 and the pre-built packages (which now know about PORT_LLVM, but have it turned off) are on v1.72.
 
putney said:
Code: 
#PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_URL=pkg+http://pkg.FreeBSD.org/\${ABI}
PACKAGE_FETCH_BLACKLIST=""
PACKAGE_FETCH_WHITELIST="gcc* rust llvm*"
Click to expand...
This is so fucked up - being forced to build HUGE things like LLVM or RUST with zero need (or sometimes resources) to do that ... Poudriere should finally allow such build - like: use whatever the fuck rust in the packages is available and do not waste time building it
.
 
vermaden said:
This is so fucked up - being forced to build HUGE things like LLVM or RUST with zero need (or sometimes resources) to do that ...
Click to expand...
Totally agreed. My first poudriere try compiled LLVM on my small lab machine and it takes 14 h. Temporal and electric energetic wastes. So, it's no way for me. I prefer to use ports. At least, you can always install what you need with pkg or make install-missing-packages.
 
Ehm, the whole point of poudriere being able to use the pre-built packages is so you don't have to build it. But you're going to have to accept the default options on that particular package. You don't need to build LLVM (or rust) if you can accept the default package for it. This doesn't mean you cannot set options on some port that has a dependency on that LLVM package. Just that you cannot change the options of LLVM itself.
 
SirDice said:
Ehm, the whole point of poudriere being able to use the pre-built packages is so you don't have to build it. But you're going to have to accept the default options on that particular package. You don't need to build LLVM (or rust) if you can accept the default package for it. This doesn't mean you cannot set options on some port that has a dependency on that LLVM package. Just that you cannot change the options of LLVM itself.
Click to expand...
I used a poudriere jail and it didn't ask my opinion on this matter. I even tried to copy inside the poudriere default repo the llvm19 package from the host, but it didn't want to use it.

After searching a few how to make it to not compile this monster, I eventually read somewhere that it's against the poudriere philosophy. It needs to compile all the dependencies no matter if no option were changed. If I'm wrong how to do it?
 
You have to tell it to use packages when possible (and those packages will need to meet the stated requirements), or else it will indeed build it from source.
poudriere-bulk(8)
Code: 
     -b branch
              Fetch binary packages from a binary package repository instead
              of building them.  The branch argument can be one of the
              following: latest, quarterly, release_X (where X is the minor
              version of a release, e.g., “0”), or url.

              With this option poudriere will first try to fetch a binary
              package from the specified binary package repository.

              poudriere will only use packages that:
              •   come from a repository having the same or older version of
                  pkg.
              •   do not have a locally fetched package already.
              •   are not IGNORED.
              •   match the expected local version.
              •   match the expected ABI.
              •   match the expected runtime and library dependencies.
              •   match the expected OPTIONS when CHECK_CHANGED_OPTIONS is
                  enabled (default: on).
              •   is NOT listed in PACKAGE_FETCH_BLACKLIST.
              •   is NOT listed with -C, or -c, when -t is used.
              The -v flag can be used to show these decisions during build.
              Specifing twice will show more details on why some are skipped.

              WARNING: poudriere has no way of determining differences outside
              of the above list.  That is, if the local ports framework, or
              port, has custom patches or special WITH_FOO knobs (not OPTIONS)
              then it is required to add its name into
              PACKAGE_FETCH_BLACKLIST.  Otherwise a package may be fetched and
              used that lacks the custom patch or knob.

              See PACKAGE_FETCH_BRANCH, PACKAGE_FETCH_URL,
              PACKAGE_FETCH_BLACKLIST, and PACKAGE_FETCH_WHITELIST in
              poudriere.conf.sample.  The entries in the lists will be matched
              against package names without versions.
At first this was only available in the -devel version, but I see the 'regular' version should have this option now too.

Personally never used it though, my old build server probably could have used it (took 5h43m to build rust, llvm15 took 04h46m) but I always let it build everything. My "new" (it's ~12 year old hardware) build server is much faster (rust; 01h48m, llvm19; 01h45m) so I'm fine with building everything from scratch.
 
SirDice

Thank you for all the things I already tried and NONE of them work ... and yes - I use poudriere-devel package.

I did not changed ANY options for LLVM/RUST/GCC/...

I use -b latest and they are still built.
 
I think that you also need to ensure that your ports tree (used by poudriere) matches the ports tree from which the upstream packages have been built.
Which is not very practical.
 
Emrion said:
If I'm wrong how to do it?
Click to expand...
I have found poudriere image a delight to work with. Sure I had to modify the scripting for ARM64 but I can reliably spit out any image I need.

Using packages versus ports for poudriere is still voodoo to me. Very spotty.
 
You must log in or register to reply here.
Back
Top