Solved Why is ports-mgmt/poudriere building lang/rust?

K

kjpetrie

This is my pkglist for ports-mgmt/poudriere:
Code: 
converters/libiconv
databases/gdbm
ftp/wget
lang/php81
mail/milter-greylist
mail/p5-Sendmail-Milter
mail/postfix
mail/postfix-policyd-spf-perl
mail/postsrsd
misc/help2man
net/netcat
net/openntpd
net/rsync
ports-mgmt/dialog4ports
ports-mgmt/pkg
ports-mgmt/portmaster
ports-mgmt/poudriere
print/indexinfo
security/clamav
security/py-certbot
shells/bash
sysutils/ezjail
textproc/expat2
www/apache24
www/mod_php81

None of my machines has lang/rust installed, not even the build machine with Poudriere on it, so why is the system wasting 3 hours every build on a language I don't want and don't use?

How can I stop it wasting my time on this useless piece of bloat?
 
Yes, I've just tracked it down. Clamav has been rewritten in Rust, it seems. This is a real threat to Free Software because if Rust takes so long to build compared with other languages it will make systems unmaintainable. The other 144 packages build in under half an hour and Rust takes a whopping 3 hours+ on my system. That's just not reasonable or bearable.

Looks as if Rust could succeed where systemd has so far failed. Perhaps developers need to be discouraged from using it.

In the meantime I will look for an alternative to Clamav.
 
Rust itself is good. From a system that mostly uses Rust, and end programs that use Rust, the software is built up better from a strong foundation. The problem is when multiple languages are used, if they're different enough, it causes problems like this.

It's a build dependency, so a dependency like that likely won't have much of a bad effect on performance. It will mostly be a problem for building. I don't use Poudriere, but for my own computer, I use portmaster which can be made to install build only dependencies through packages. If using packages for build only dependencies (either manually for large programs like Rust or with portmaster) can be incorporated into Poudriere with confidence that it won't affect your build, go for that.

The problem with Rust for many, especially in the past for those using Firefox, was that one dependency asked for a nightly build of Rust every time, and there wasn't a package on time as soon as the latest dependency was needed. They fixed that, by making it reliant on Meson and maybe a more stable version of Rust also.
 
FreeBSD has an enduring problem with Rust requiring COMPAT_FREEBSD11

⚙ D23100 lang/rust: Add COMPAT_FREEBSD11 canary

reviews.freebsd.org reviews.freebsd.org

and the Rustians are ignoring this ever since. There are PRs on this and the port maintainer is getting somehow annoyed about further incoming PRs.

Trying to build ports like Rust with Poudriere is such a waste of time and energy that it is getting more and more difficult to channel emotions for the better.

On the other hand Poudriere can fetch packages from reposities (edit poudriere.conf if neccessary) with poudriere bulk -b latest lang/rust. But this requires that you stay with the default port options downside Rust's dependencies and of course that the Rust-package is provided when you need it.
 
For now, I've swapped security/clamav for security/clamav-lts. That should last another year. I'll look into the -b option as well. Obviously, spending 6 times as long to build one program's build-time dependency as it takes to build everything else is unacceptable.

I'd be fed up if I were the maintainer of such a monster. Apparently Rust's developers are aware it takes too long to compile so let's hope they tidy it up before long. Part of the problem is that the compiler is written in itself, so it has to do multiple passes to build itself.
 
Rust may be more suitable for its own ecosystem. A lot should be forked from having the last version that was written in C and a different version of Rust. In a Rust ecosystem, it makes more sense to have Rust and C together. FreeBSD is a ecosystem of a few languages, but it's primarily C and C++, with the addition of scripting languages.
kjpetrie said:
Part of the problem is that the compiler is written in itself
Click to expand...
On FreeBSD, it uses LLVM which is written in C++ the last time I checked. Another problem is that on FreeBSD, CLANG isn't treated in a modular way from LLVM. Rust is treated modularly from LLVM and Clang though.

The problem of perpetual dependency hell just for 1 library, which was all that was needed, used to be a lot worse on FreeBSD. It was a chain reaction of dependencies for 1 simple option or dependency. The problem wasn't just from build dependencies though.

A lot of programs which have build dependencies of multiple programming languages causes dependency problems. For instance, libcanberra is also written in Vala.
 
kjpetrie said:
Yes, I've just tracked it down. Clamav has been rewritten in Rust, it seems. This is a real threat to Free Software because if Rust takes so long to build compared with other languages it will make systems unmaintainable. The other 144 packages build in under half an hour and Rust takes a whopping 3 hours+ on my system. That's just not reasonable or bearable.
Click to expand...
I don't think rust should be used "just because" everywhere. There's (so far) only one implementation, no independently developed language standard (like with C and C++), problems with this specific implementation (see above), and so on. For ClamAV, I really don't get it.

Still, comparing build times like this makes little sense. To be fair, you'd have to add the build time for llvm or gcc to your other packages.

Who knows, maybe rust will see an independent standard committee and competing implementations some day? And maybe by then, a rustc in a base system will be just as natural as cc (and later cxx)?
 
kjpetrie said:
Part of the problem is that the compiler is written in itself, so it has to do multiple passes to build itself.
Click to expand...
Which is true for any serious compiler. It's (among other things) a quality check for the language as well as the implementation. A full build of gcc or llvm also does multiple passes.

Once a language is widespread enough, so you can reasonably assume anyone who wants to build an implementation has another implementation ready (this is true at least for C), "foreign" languages can be removed. The multiple passes will stay. A complete scheme could look like this:
  1. Build the compiler using whatever is present on the host machine
  2. Build the compiler using the result of step (1)
  3. Build the compiler using the result of step (2)
  4. Compare results of step (2) and (3), they must be identical
 
The practical reality many system administrators will face will be they can't afford the CPU time it takes to build this monster just for Clamav. I've put this on their bugzilla:
When bulk-building packages for Linux repositories or FreeBSD servers, processor time is precious and we cannot afford the 3-6 hours it takes to build Rust before we can build Clamav. It is therefore imperative, if Clamav is to continue to be useful on such platforms, that a version which is not reliant on Rust should remain available.


That either means extending the life of version 103, currently supported for another year, or removing the build dependency in future versions. Otherwise this huge overhead will mean Clamav will have to be abandoned by system administrators as too expensive to maintain on their systems.


It's not so much a bug as a practical reality.
Click to expand...
I expect they'll just close it "won't fix", but the result will probably be mail server administrators closing Clamav as "won't install".

For now,it's at https://github.com/Cisco-Talos/clamav/issues/677#issuecomment-1211907511
 
This most likely won't lead anywhere. Almost any serious "production server" will use binary packages. And even if you have, for some reason, your own build server in your infrastructure, it will be a "beefy" machine anyways, and it most certainly will have to build rust for other software written in it.

To take it to an extreme: You could issue a similar complaint about having to build llvm (or gcc). After all, upstream could just write code in assembler.

As I said, currently, rust is not to be considered "standard" (as in, every system building any software will need it anyways). Therefore, you should have good reasons to rewrite existing code in rust. But the amount of software written in rust is ever increasing, I don't think you can stop that by writing such a complaint.
 
zirias@ said:
Still, comparing build times like this makes little sense. To be fair, you'd have to add the build time for llvm or gcc to your other packages.
Click to expand...
gcc is part of the base system, not a port which has to be built for just one package, so it's not really a fair comparison. If you put Rust in the base system the problem would disappear at the expense of the base system becoming bloated.
Click to expand...
 
kjpetrie said:
This is my pkglist for ports-mgmt/poudriere:
Code: 
converters/libiconv
databases/gdbm
ftp/wget
lang/php81
mail/milter-greylist
mail/p5-Sendmail-Milter
mail/postfix
mail/postfix-policyd-spf-perl
mail/postsrsd
misc/help2man
net/netcat
net/openntpd
net/rsync
ports-mgmt/dialog4ports
ports-mgmt/pkg
ports-mgmt/portmaster
ports-mgmt/poudriere
print/indexinfo
security/clamav
security/py-certbot
shells/bash
sysutils/ezjail
textproc/expat2

None of my machines has lang/rust installed, not even the build machine with Poudriere on it, so why is the system wasting 3 hours every build on a language I don't want and don't use?

How can I stop it wasting my time on this useless piece of bloat?
Click to expand...
To solve this problem you need to run the following:

Code: 
cat | xargs ag -G Makefile rust
converters/libiconv
databases/gdbm
ftp/wget
lang/php81
mail/milter-greylist
mail/p5-Sendmail-Milter
mail/postfix
mail/postfix-policyd-spf-perl
mail/postsrsd
misc/help2man
net/netcat
net/openntpd
net/rsync
ports-mgmt/dialog4ports
ports-mgmt/pkg
ports-mgmt/portmaster
ports-mgmt/poudriere
print/indexinfo
security/clamav
security/py-certbot
shells/bash
sysutils/ezjail
textproc/expat2
<ctrl-D>

The output of which is:

Code: 
security/clamav/Makefile
13:BUILD_DEPENDS=    ${RUST_DEFAULT}>=1.56.0:lang/${RUST_DEFAULT} \
 
zirias@ said:
This most likely won't lead anywhere. Almost any serious "production server" will use binary packages. And even if you have, for some reason, your own build server in your infrastructure, it will be a "beefy" machine anyways, and it most certainly will have to build rust for other software written in it.
Click to expand...
I don't have a "dog in this game of rust", but I find this interesting. I agree with it, it is basically the same position that OpenBSD has with reporting of bugs, especially in the kernel or base. "Did this happen with GENERIC or a custom kernel? If custom retry with GENERIC and then get back to us". Almost "If it was not with GENERIC it never happened".
Folks can argue if that is a reasonable stance or not, but it is consistent.
Dedicated Build Servers are often beefy machines simply because they are dedicated to one task and that task has clearly defined requirements.
Anyone complaining about a 3hr Rust build, well, we won't talk about overnight Ada builds on a DEC Vax.
 
I reviewed my mail logs and found no mail was reaching Clamav since other spam filter techniques (Greylisting and Blocklists) prevented virus mail getting that far.

In the light of that, the remedy was obvious! I have removed security/clamav-lts from my machines and from ports-mgmt/poudriere's pkglist.

Not that surprising really. When one thinks about it, virus mail is just another variety of spam and shares its characteristics.

Update: Cisco's response contains an interesting comment:
...C was great in its time but has not aged well. The Unix/Linux package management model is essentially a C library package management ecosystem. Rust is certainly not the first of newer languages to find it difficult to work with legacy C/Unix package distribution models.
Click to expand...
Are they throwing down a gauntlet - suggesting *nix is out-of-date?
 
That Update with Cisco's response. Man. Why is C then the "standard" everything is compared to?
Rust "package management" are we saying that the nodejs (npm), python and everything else with their own package management is the right model?
Or that everything should be in it's own container, with all the dependencies instead of shared libraries?
Or that everything should be statically link instead of dynamically linked?

I'm not saying "Rust is bad or a bad idea" I think it's actually not bad but often theory runs into the real world. Until "everything is Rust" it will always have to interface or work with legacy, same thing as every other next big thing language has/will have to do.

Jose Awesome. I think we've all run into that.
 
I personally think this response is extraordinarily stupid.

I expected the request to be refused, as explained above. And I don't see anything bad with rust per se.

But talking about "C library package management ecosystems" is just ridiculous. This tendency now to create language-specific package managers is ridiculous as well. But as far as I can see, software written in rust plays nice with your typical generic package management. So, they completely missed the point and made something up that just sounds stupid.
 
From what I've been seeing on Rust is that it has a good idea; and works fairly well. The issue is that it is just doing the "follow the leader" on package management blindly without considering any of the issues. If anything, it is more of rust is going to end up crashing and burning iin the end as it is going. The reason is that they are diving head first into the package management for everything. It doesn't help in that they are resistant on having a standard core library; and rather move anything that is in there core library out of it into 3rd party packages; so they don't have to deal with it. I've seen discussions, on the dependency h***, that is so common on package management, is that they'd rather sweep it under the rug. It seems the only reasons rust hasn't been hit much (or at least not publicly) is that it's still "young", to where the package documentation is still lacking that you usualy resort to going through the source code to figure out how stuff works (so more likely to catch questionable code).
 
  • Like
Reactions: mer
A response provided over at GitHub:
Rust is the language that our team enjoys now, and that we have a lot of confidence in. I am looking forward to the future where we will do more and more in Rust. Given the size of this project and the size of our team, I'm not sure if we can ever put C completely behind us, but I can cross my fingers.


The new fuzzy image hash detection mechanism (one of two ClamAV features now implemented in Rust) is quite important to us. Further, C was great in its time but has not aged well. The Unix/Linux package management model is essentially a C library package management ecosystem. Rust is certainly not the first of newer languages to find it difficult to work with legacy C/Unix package distribution models.


OpenSUSE is one distribution that is putting a lot of thought into how to package Rust and Rust-based applications. The Rust package maintainer for OpenSUSE gave a pretty good talk about it at RustConf last week. I wish the video were available to share now, but it is still private for conference attendees only.
Click to expand...
 
I take the package management thing to mean they want some library without having it being bunch of work to use it.
 
You must log in or register to reply here.
Back
Top