How to go with FLAVOR?

PMc

Aspiring Daemon

Reaction score: 179
Messages: 511

Hi all, I've seen there was already some discussion about the FLAVOR feature. Now it seems to hit me. :(

Somewhere ShelLuser wrote:
I don't understand the confusion. [...] See, the whole FLAVOR thing is merely a make specification. Nothing different from, say, BUILD_OPTIMIZED=yes or DISABLE_VULNERABILITIES=yes.

And if you need to use those permanently, then simply define them globally. So: using /etc/make.conf. Problem solved. ((edit): note that I don't recommend setting DISABLE_VULNERABILITIES within /etc/make.conf).
It's not quite that easy. Indeed it is quite difficult.

When I update the portstree (once a year, or so), I do the following [1]:
1. List all ports installed on a system.
2. Get all dependencies for all of these ports (make all-depends-list)
3. Sort all these dependencies into a hierarchy (what depends on what?)
4. Build the whole stuff in that proper sequence.
5. Update the respective system from the resulting packages, and all proper dependencies are now available.
This worlks as trouble-free as installing from packages, but it works with my settings in make.conf and with my patches in the ports. ;)

But now, e.g. firefox tells me it somehow depends on [PMAN=]devel/py-setuptools[/PMAN]. So I ask make describe for the proper name to build, and I get four answers:
Code:
py27-setuptools devel/py-setuptools
py34-setuptools devel/py-setuptools
py35-setuptools devel/py-setuptools
py36-setuptools devel/py-setuptools
Now, I could proably declare in make.conf "I want to go with (e.g.) py35, only". But that does not work, because firefox depends on some pieces that need python2 and some that need python3.
In the past this was sorted out, because each of these pieces would declare it's own dependencies uniquely. Now they all declare "devel/py-setuptools", and it is ambigous, and I don't see how that could be sorted out, except manually for each case.

I could also build all four of them, but there is no point, because some of them are probably conflicting to each other - they can be individually built, but not installed together before building firefox.

I suppose the FLAVOR thing is a good thing, and it can solve problems. (Indeed, the creepy "slave ports" were a nuisance.) But at this point, it seems, it blows up my scheme altogether, and currently I don't see how to devise a new scheme. :(

[1] I do this partly with portupgrade and partly with own scripts (the parts where portupgrade has not proven reliable enough to cleanly rebuild a whole desktop and me to go for a sleep in the meantime). Probably, meanwhile I could drop portupgrade alltogether and do it just with make.
 

ShelLuser

Son of Beastie

Reaction score: 1,749
Messages: 3,565

When I update the portstree (once a year, or so), I do the following [1]:
Wait, so you're telling me that you're only updating your system once every year? Then I think you found the source of your problems. No matter what kind of updater you use that really isn't the best of ideas, especially not if this involves a server which is also connected to the Internet.

Even so... I've been somewhat in the same situation with my laptop (updated it once every 3 - 4 months) but never had any issues. When done right the long period doesn't have to be a problem, same applies to flavors.

1. List all ports installed on a system.
2. Get all dependencies for all of these ports (make all-depends-list)
3. Sort all these dependencies into a hierarchy (what depends on what?)
4. Build the whole stuff in that proper sequence.
Sounds to me as if you're making things way too hard on yourself. Just let the system sort the whole thing out, this is also why you have tools such as ports-mgmt/portmaster. Your method isn't necessarily wrong of course, I can definitely understand why you'd prefer to keep full control over your building process, but that also requires you to keep yourself informed about the current and new standards and making sure that you understand how they work.

But now, e.g. firefox tells me it somehow depends on [PMAN=]devel/py-setuptools[/PMAN]. So I ask make describe for the proper name to build, and I get four answers:
Code:
py27-setuptools devel/py-setuptools
py34-setuptools devel/py-setuptools
py35-setuptools devel/py-setuptools
py36-setuptools devel/py-setuptools
And this is what I meant with learning how things work. You're using the wrong commands. I also happen to have FireFox installed on my server / (sporadic) workstation and guess what:

Code:
peter@unicron:/usr/ports/www/firefox $ make describe | grep setuptools
peter@unicron:/usr/ports/www/firefox $ make all-depends-list | grep setuptools
/usr/ports/devel/py-setuptools
/usr/ports/devel/py-setuptools_scm
Of course I have a DEFAULT_VERSIONS entry set up in /etc/make.conf but that's really besides the point. Because without that entry the system will automatically use the current default. If you check /usr/ports/Mk/bsd.default-versions.mk you'll see the current defaults. For Python this is:
Code:
$ grep -i python bsd.default-versions.mk
        MYSQL PERL5 PGSQL PHP PYTHON PYTHON2 PYTHON3 RUBY SSL TCLTK
PYTHON_DEFAULT?=        2.7
PYTHON2_DEFAULT?=       2.7
PYTHON3_DEFAULT?=       3.6
This is what flavors is all about: a port such as devel/py-setuptools can now provide support for a multitude of Python versions. There are no 4 versions required: only 1 while specifying the right flavor (aka version).

Now, I could proably declare in make.conf "I want to go with (e.g.) py35, only". But that does not work, because firefox depends on some pieces that need python2 and some that need python3.
See above. This is why there are currently 2 versions of Python being maintained next to each other.

In the past this was sorted out, because each of these pieces would declare it's own dependencies uniquely. Now they all declare "devel/py-setuptools", and it is ambigous, and I don't see how that could be sorted out, except manually for each case.
Simple: just let the ports sort it out.

As far as I know the flavor also follows the default version. So if you go into a 'flavored' port directory and issue the regular building commands then the result would be a port for that specific default version. To my knowledge it should be possible to override this on the commandline, but so far I never really bothered myself to look deeper into that. I usually let the ports building process sort this out for me.

Which is what I would recommend doing. If you build www/firefox then it will also automatically set up the build and run dependencies (see: # make build-depends-list and # make run-depends-list respectfully). Why not simply rely on that?

Or use a simple tool such as ports-mgmt/portmaster which can automate those parts for you while it still leaves you in full control. For example, if all you want to do is build FireFox (and its dependencies) as packages which you can then deploy to other servers you could use something such as: # portmaster -g www/firefox.

Hope this can give you some ideas.
 
OP
OP
PMc

PMc

Aspiring Daemon

Reaction score: 179
Messages: 511

Wait, so you're telling me that you're only updating your system once every year? Then I think you found the source of your problems. No matter what kind of updater you use that really isn't the best of ideas, especially not if this involves a server which is also connected to the Internet.
Okay, You are probably right from an official viewpoint.
But, well, to me it appears a year is happening quite often (maybe I'm getting old). And, I have a long-term aim (which actually never really happended to fulfil, but nevertheless): the reboot happens at christmas.

Then, there is a saying: It's cheaper to move big chunks. And there is another saying: change only one parameter. So, if You update often, you have continuousely random parametes changing, and you don't even know what has changed without researching.
So, concerning You point, that would make sense for dedicated production system with well-defined use-cases, where there is also an integration system in place with automated tests for these use-cases. Then you can implement a cyclic scheme of frequent updates.
Otherwise, if it is an individual multi-purpose system, one must consider if one wants things to be stable, so to be able to figure out the effects of one's own recent wrongdoings, or if one wants to be drowned in things constantly changing, but then be up-to-date.

Sounds to me as if you're making things way too hard on yourself. Just let the system sort the whole thing out, this is also why you have tools such as ports-mgmt/portmaster. Your method isn't necessarily wrong of course, I can definitely understand why you'd prefer to keep full control over your building process, but that also requires you to keep yourself informed about the current and new standards and making sure that you understand how they work.
Oh, don't think I didn't try that!
My own scripting had the sole purpose to workaround issues where the "system deciding" produced crap. Two examples:
* Openoffice builds differently, if the build-dependency of one of its dependencies is not deinstalled before OO itself is built. (Surely, both versions do work - but I prefer defined behaviour.)
* Some graphics library checked for another library (used for building something different), and if present, would not create a dependency to it, but would instead write it into its own elf header! Obviousely, on the target system that library was not present, and I had a hard time working myself through a vast tree of interdependent sharedlibs to find the issue.

I do not know if these specific effects do still exist, but I met quite a couple of those over the years. For that reason I got so bothered that I decided to do it properly, i.e. use switching zfs snapshots where the port to be built can only see exactly what does concern it and does not get the chance to grab something else.

Code:
peter@unicron:/usr/ports/www/firefox $ make describe | grep setuptools
peter@unicron:/usr/ports/www/firefox $ make all-depends-list | grep setuptools
/usr/ports/devel/py-setuptools
/usr/ports/devel/py-setuptools_scm
Ups, sorry, but thats not quite what I am doing. Rather this:
Code:
@/usr/ports/www/firefox # make all-depends-list | grep setuptools
/usr/ports/devel/py-setuptools
/usr/ports/devel/py-setuptools_scm
@/usr/ports/www/firefox # cd /usr/ports/devel/py-setuptools
/usr/ports/devel/py-setuptools # make describe | \
    awk -F"|" '{print $1, $2}'
py27-setuptools-39.0.1 /usr/ports/devel/py-setuptools
py36-setuptools-39.0.1 /usr/ports/devel/py-setuptools
py35-setuptools-39.0.1 /usr/ports/devel/py-setuptools
py34-setuptools-39.0.1 /usr/ports/devel/py-setuptools
Of course I have a DEFAULT_VERSIONS entry set up in /etc/make.conf but that's really besides the point.
Yes, that doesn't solve the issue. Currently I do not have a default version, because I am investigating how the defaults that come with this tree behave and how far I can live with that.

This is what flavors is all about: a port such as devel/py-setuptools can now provide support for a multitude of Python versions. There are no 4 versions required: only 1 while specifying the right flavor (aka version).
That is exactly what I am talking about! I need to specify the right flavor - but it is not me who wants this py-setuptools ports, it is firefox! So firefox needs to somehow tell me first which flavor it wants, so that I can specify that.

Simple: just let the ports sort it out.
Thanks, but no thanks. See above.

In the meantime I found an undocumented switch, which appears to solve the issue (it wouldn't be FreeBSD if there weren't a solution ;) ):
Code:
@/usr/ports/www/firefox # make -DDEPENDS_SHOW_FLAVOR all-depends-list | grep setuptools
/usr/ports/devel/py-setuptools@py27
/usr/ports/devel/py-setuptools_scm@py27
Voila! Now firefox tells me what I need to know. This appendix can be given to the py-setuptools port, and while make describe does not honor that option (and still provides four answers), the respective values can be queried as make variables directly:

Code:
@/usr/ports/devel/py-setuptools # make FLAVOR=py27 -V PKGNAME
py27-setuptools-39.0.1
@/usr/ports/devel/py-setuptools # make FLAVOR=py34 -V PKGNAME
py34-setuptools-39.0.1
I suppose this might be enough to get along. One needs to consider that there are now three tuples to identify a port: origin, version and flavor. If this works out, it should indeed make things overall better.

Which is what I would recommend doing. If you build www/firefox then it will also automatically set up the build and run dependencies (see: # make build-depends-list and # make run-depends-list respectfully). Why not simply rely on that?
Well, honestly, because the "sendbug" command has disappered. About once a year i fetch a new portstree, and then it's about a week of vermin hunting.
Those ports makefiles are written by lots of different people, and they make certain assumptions about what the user might want. But I don't want assumptions being made, I don't want AI. I want a system to do precisely what is requested - and if the request is vague, it should fail with an error (instead of doing something vague that might work most of the time).
 

drhowarddrfine

Son of Beastie

Reaction score: 1,283
Messages: 3,330

You are probably right from an official viewpoint.
Many of us update ports daily. Maybe all of us. Some through a cron job, I don't know.
So, if You update often, you have continuousely random parametes changing, and you don't even know what has changed without researching.
That's why you read /usr/ports/UPDATING but continuous parameters changing? Not hardly so.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,021
Messages: 31,497

And, I have a long-term aim (which actually never really happended to fulfil, but nevertheless): the reboot happens at christmas.
Uptimes are overrated.

Why don't you switch to a quarterly branch? Then you only need to update once every three months or for security issues (keep an eye on VuXML; pkg-audit(8)). Updating once a year is a recipe for disaster.
 
  • Thanks
Reactions: PMc
OP
OP
PMc

PMc

Aspiring Daemon

Reaction score: 179
Messages: 511

Uptimes are overrated.
Yes, but: reboot is not a solution. ;)

Why don't you switch to a quarterly branch?
Ah, we have stable portstree branches now! Didn't notice that. Thats indeed nice and makes life easier. (Usually the portstree always changes while building from it, so if you accidentally delete a file, you cannot simply fetch it again from SVN without giving the original revision number.)
 

Trihexagonal

Daemon

Reaction score: 992
Messages: 1,654

Wait, so you're telling me that you're only updating your system once every year?
Those are the exact words that went through my mind as I read it. I suggest running pkg audit -F first next time you do to see how many vulnerabilities your server has been subject to over the past year, out of curiosity.

If you're only going to update your ports tree and programs once a year, why not just go with the next version of FreeBSD, build them all fresh and be done with it? Despite downtime you would be ahead in the game IMO.
 
OP
OP
PMc

PMc

Aspiring Daemon

Reaction score: 179
Messages: 511

Those are the exact words that went through my mind as I read it. I suggest running pkg audit -F
Well, I think that one is a new feature that did appear with the pkg stuff. Wasn't there before.

first next time you do to see how many vulnerabilities your server has been subject to over the past year,
And that tells You what? Have You ever bothered to check those "vulnerabilities" and figure which attack vectors would be relevant for the specific installation? Or otherwise, is this rather about a psychological condition named hysteria?

I remember times when in the more qualified discussion groups most of the people had their own "vulnerability" at disposal, so they could get access to an otherwise locked FreeBSD system. Those were quite good times.

Or, lets put it otherwise: why should I be worried about a list of already published "vulnerabilities", while anyway I can't do much about those "vulnerabilities" that are still in use and therefore not yet being published?
 

talsamon

Daemon

Reaction score: 281
Messages: 1,822

But you can reduce the number of "possible" vulnerabilities.
And I doubt you have a better "overview" if you update once a year. I think you have a better, if you do it in an interval of e.g. few weeks.
------
If I imagine this:
Either you update with ports-mgmt/portmaster, the order of the ports amd its dependencies will never be right. Portmaster will interrupt one-hundred times. If you do it with ports-mgmt/poudriere, it will compile an eternity and one day (but I think, the latter would be the better way).
 

talsamon

Daemon

Reaction score: 281
Messages: 1,822

627 packages is a "minimal-install". I guess without webkit\* and any compiler.
 

rigoletto@

Daemon
Developer

Reaction score: 1,037
Messages: 2,097

This is a desktop, but yes I do not use anything that need webkit, but compilers are built as necessary (just not installed) - also rust.

I just forgot to say that time is using devel/ccache, without that take a about 6h30m (IIRC).
 

talsamon

Daemon

Reaction score: 281
Messages: 1,822

I don't believe that. Poudriere build per default with one builder, with one builder a compiler needs one and a half hour. If you set bigger packages to ALLOW_MAKE_JOBS_PACKAGES and two big packages builds at the same time it slows ugly down. (This is a big problem with
poudriere - see PR 228364).
 

rigoletto@

Daemon
Developer

Reaction score: 1,037
Messages: 2,097

Why stuck with defaults?

Code:
PARALLEL_JOBS=3
PREPARE_PARALLEL_JOBS=5
AMD FX-8020, 16GB RAM. That is the fastest combination I found, and if I change any of those numbers the build time increase at least 2 hours.

Btw, I do not use ALLOW_MAKE_JOBS_PACKAGES anymore, but had no problem using in the past.
 

talsamon

Daemon

Reaction score: 281
Messages: 1,822

My guess was far away:
Code:
[104amd64-default] [2018-06-08_08h45m02s] [ ===> parallel_build:] Queued: 1  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 1   Time: 02:42:17
    [01]: lang/gcc6                 | gcc6-6.4.0_7              package         (00:00:52 / 02:41:53)
[02:42:23] Logs: /usr/local/poudriere/data/logs/bulk/104amd64-default/2018-06-08_08h45m02s
===> [02:46:47] [01] [02:46:17] Finished lang/gcc6 | gcc6-6.4.0_7: Success
Ok, I forget to close the browser, but I think this will only half the time.
 

rigoletto@

Daemon
Developer

Reaction score: 1,037
Messages: 2,097

I just left it running in here while doing my stuff (browsing etc.).

Untitled.png
 
Top