Synth version 2.00: Flavor support

After getting blindsided with the release of flavors, I spent a few hours upgrading Synth to support them. You can upgrade to the latest version of Synth immediately as follows:

  • cd /usr/ports/ports-mgmt/synth
  • edit Makefile, change DISTVERSION from "1.71" to "2.00"
  • make makesum
  • make clean deinstall
  • make install

There is one major change to how Synth works now. In order to handle the change from 1-to-1 to many-to-1 relationship between packages and ports, a new index is required. Since ports doesn't supply this index, Synth has to generate it whenever the ports tree changes. To do this, Synth has to scan every single port in the tree. On DragonFly, this takes about 60 seconds. On FreeBSD, it takes several minutes. (The technical explanation is out of scope, but basically dports fixes several bottlenecks that never made it back to freebsd ports).

So when new new Synth starts up, it will immediately begin the scan. Subsequent launches of Synth will work as before until the ports tree is updated.

For ports developers:
This new behavior can cause problems if you're modifying a port. Synth will detect if a single file is newer than the current index. For you, I've added the following logic: As long as the ports tree directories match the new index (no port directories added or deleted), modifying a file will result in this message:

Code:
*******************************************************************
 The ports tree has at least one file newer than the flavor index.
 However, port directories perfectly match.  Should the index be
 regenerated? (Y/N)

Normally you'd select "N" which would "touch" the index and continue without performing a lengthy scan. If your modifications define or remove flavors, you may want answer "yes" here.

Regular users will not see this message though, and will only have to deal with the new index generation when the port tree changes.

Hopefully tobik@ will find this post and update ports soon. The synth port maintainer already requested the update (as you could imagine).

I've done some testing and everything seems to work fine, but this version represents a LOT of new code so please report any strange behavior if you find any.

-- John
 
Oh, I forgot to mention another new behavior.
If a port features flavors, but you don't specify any specific favor, the framework picks the first listed flavor. This is considered the default.
So lets take meld as an example. In theory, you should be able to specify "textproc/meld" and synth will build it with the default flavor. I decided to make the input really explicit, so Synth rejects flavored ports with no specified flavor. Examples:

synth synth force textproc/meld
Code:
Error: port origin 'textproc/meld' is not recognized.
Perhaps you intended one of the following port flavors?
   - textproc/meld@py27

You will get suggestions if the flavor is missing or incorrect

synth force accessibility/py-atspi@py28
Code:
Error: port origin 'accessibility/py-atspi@py28' is not recognized.
Perhaps you intended one of the following port flavors?
   - accessibility/py-atspi@py27
   - accessibility/py-atspi@py36

This may cause some existing build lists to be modified. pkg(8) will no longer provide good build lists out of the box
 
Hmm, I just realized that synth status (and related) is probably not going to work anymore.
The pkg query %o command only returns origins (without flavors). This needs to be modified to have a per-package query for annotations to try to find flavors. That will have to come in later versions.
For now, synth status is broken and you'll have to update manually.
 
After doing all the updates, this shows up if I do a synth prepare-system

I believe most of these packages were just updated to the new py27 flavor. Other than that, everything works and looks great.

Code:
$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of devel/py-babel failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of textproc/py-docutils failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of textproc/py-pystemmer failed, it will not be considered.
Scan of devel/py-six failed, it will not be considered.
Scan of textproc/py-sphinx_rtd_theme failed, it will not be considered.
Scan of textproc/py-MarkupSafe failed, it will not be considered.
Scan of devel/py-pytz failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of textproc/py-pygments failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of devel/scons failed, it will not be considered.
Scan of textproc/py-alabaster failed, it will not be considered.
Scan of devel/py-Jinja2 failed, it will not be considered.
Scan of textproc/py-sphinx failed, it will not be considered.
Scan of textproc/py-snowballstemmer failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scan of graphics/py-imagesize failed, it will not be considered.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
 
Is pkg(8) going to take care of this
Code:
DEFAULT_VERSIONS   +=   mysql=5.6 ssl=openssl
in usr/local/etc/synth/LiveSystem-make.conf?
 
Last edited by a moderator:
After doing all the updates, this shows up if I do a synth prepare-system

I believe most of these packages were just updated to the new py27 flavor. Other than that, everything works and looks great.

As I said, synth status and related was anticipated to be broken. synth prepare-system is just an extension of synth status. It's also just a convenience command. You can use a build list to build everything, including python (until it's fixed).
In fact, a build list is a better way to maintain the system.
 
Is pkg8 going to take care of this
Code:
DEFAULT_VERSIONS   +=   mysql=5.6 ssl=openssl
in usr/local/etc/synth/LiveSystem-make.conf?

I will be reflected in the repository that you build, which pkg(8) respects (assuming you're not mixing repos or you turned off CONSERVATIVE_UPGRADE and put your local repository as the higher priority)
 
marino
OT: Yesterday I read a lot. It was very interesting to learn how you were thrown out of the FreeBSD team. Let me just say that I find it very sad how unoutspoken butthurt probably caused by narcissistic immaturities of some people led to the expulsion of an essential FreeBSD developer from the team.
So I am very glad that you seem mature and do not take immature of immature people personally and keep contributing to the FreeBSD community. (I refrain from going into detail, because this is obviously emotionally very sensitive to some people.)

Back to topic: As it looks like that my impression of Synth development being stopped was wrong, I will give Synth a try the next days.

pkg(8) will no longer provide good build lists out of the box
Thus I'd like to ask, is there a (simple) pkg command to rebuild all its lists to reflect the current system state etc if something goes wrong when playing with Synth?
 
Hmm, I just realized that synth status (and related) is probably not going to work anymore.
The pkg query %o command only returns origins (without flavors). This needs to be modified to have a per-package query for annotations to try to find flavors. That will have to come in later versions.
For now, synth status is broken and you'll have to update manually.

Thank you very much marino!

When you mention that you'll have to update manually, do you mean you have to 'synth just-build' port by port? Usually I feed the file with the list of ports to synth, but I get an error about port origin being not recognized:

Code:
# synth just-build /usr/local/etc/synth/ports.2.build
Error: port origin 'converters/libiconv' is not recognized.

Feeding converters/libiconv seems to work fine:

Code:
# synth just-build converters/libiconv
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
 
Thank you very much marino!
When you mention that you'll have to update manually, do you mean you have to 'synth just-build' port by port? Usually I feed the file with the list of ports to synth, but I get an error about port origin being not recognized:

Hmm, that message about converters/libiconv not being recognized seems like a bug. It if works via command line, it should work via file. I'll try to reproduce locally while I work on v2.01
 
Just a heads up, I'm having the same issue as nov1c3
Code:
$ sudo synth just-build build.list 
Error: port origin 'ports-mgmt/synth' is not recognized.
 
And we're off and building. I had to use synth just-build `cat build.list` rather than synth just-build build.list -- otherwise I get the port origin not recognized issue above -- and add flavors as appropriate, but now we're cooking. (And, for reference, pkg query -e '%a = 0' %o > build.list to populate the list to begin with...)

Thanks again for the great tool!!
 
Eric A. Borisch

You could use pkg prime-origins > build.list instead of pkg query -e '%a = 0' %o > build.list. ;)

Except that it won't work correctly anymore now that flavors is out.
To be clear, it won't work on flavored ports. One origin, many ports.
Since I decided to make the flavor explicit (rather than default to first flavor), the origin alone isn't enough anymore.
 
Thoughts on the flavored ports issue:
  • Could Synth, rather than skipping a flavored port that doesn't have a flavor specified, stop and ask which flavor to use?
  • Could Synth maintain a local database of historic flavor selections, so if it knows you want a certain flavor, use that flavor again unless otherwise specified?
 
Could Synth, rather than skipping a flavored port that doesn't have a flavor specified, stop and ask which flavor to use?

It doesn't skip now. It stops with an error message (and often a suggestion)

Could Synth maintain a local database of historic flavor selections, so if it knows you want a certain flavor, use that flavor again unless otherwise specified?
no. flavors must be explicitly specified. anything else is asking for problems, especially when the previous selection changes (which it inevitably will over time).
 
It's not too bad to rebuild an index on SSD's, but it sure takes a while on standard HDD. I realize that is beyond your control, but you'd think FreeBSD would have better tools in place so this wouldn't be necessary.

Is this how ports-mgmt/poudriere works as well to handle flavors? Building an index every time the port changes?
 
Using Synth 2.01, this is still showing up each time synth prepare-system is run. Is this normal?

Code:
$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 151 ports serially.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of devel/py-babel failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of textproc/py-docutils failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of textproc/py-pystemmer failed, it will not be considered.
Scan of devel/py-six failed, it will not be considered.
Scan of textproc/py-sphinx_rtd_theme failed, it will not be considered.
Scan of textproc/py-MarkupSafe failed, it will not be considered.
Scan of devel/py-pytz failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of textproc/py-pygments failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of devel/scons failed, it will not be considered.
Scan of textproc/py-alabaster failed, it will not be considered.
Scan of devel/py-Jinja2 failed, it will not be considered.
Scan of textproc/py-sphinx failed, it will not be considered.
Scan of textproc/py-snowballstemmer failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scan of graphics/py-imagesize failed, it will not be considered.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
 
Code:
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.01
====================================================================
               Copyright (C) 2015-2017 John R. Marino

----

synth-2.01                         =   up-to-date with index
 
try synth status instead, and look in the logs directory for 06* log for errors.
What you're showing is not normal. Without evidence, I'd suspect you've got something in your profile's make.conf that's breaking things.
 
Back
Top