Synth version 2.00: Flavor support

Ok, that seems to work just fine.
Code: 
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt
/usr/local/etc/synth/LiveSystem-make.conf
Code: 
OPTIONS_UNSET = X11 CUPS

WITH_POSTFIX = yes

DEFAULT_VERSIONS += ssl=openssl php=7.1 mysql=5.6
/var/log/synth/06_obsolete_packages.log is empty
 
Working for me. That is, I didn't get all the python related package errors, and I see it's already built some of them, for example getmail@py27
 
ah, it looks like the repository rebuilding is what is emitting the errors. hang on.
 
marino said:
synth prepare-system is just an extension of synth status. It's also just a convenience command.
Click to expand...

Maybe, but I see it more than that John and I use as my primary update command. Then when all has built successfully and I am ready to actually update my programs I then do my update. Anyway, thanks a bunch for your effort with this, it will likely be a few weeks before I do any updating on my main servers, but I do have a couple lab servers if you need extra 'eyes' to help test something.
 
Getting close to releasing 2.02, but I need to finish testing locally. One typo caused about 350 current packages in my repository to get deleted, so I'm building them back up. However, I'm hopeful the current synth code is good.

If anyone is feeling adventurous, they can try it in the meantime. I'd love the feedback.
to build, add the line
GH_TAGNAME= f751b76
to /usr/ports/ports-mgmt/synth/Makefile

then cd /usr/ports/ports-mgmt/synth && make makesum
then make clean deinstall
then make install


To clarify, this version tries to fix the synth rebuild-repository and synth prepare-system and similar commands. The flavor stuff really threw a wrench in the works on those.
 
marino said:
Getting close to releasing 2.02, but I need to finish testing locally. One typo caused about 350 current packages in my repository to get deleted, so I'm building them back up. However, I'm hopeful the current synth code is good.

If anyone is feeling adventurous, they can try it in the meantime. I'd love the feedback.
to build, add the line
GH_TAGNAME= f751b76
to /usr/ports/ports-mgmt/synth/Makefile

then cd /usr/ports/ports-mgmt/synth && make makesum
then make clean deinstall
then make install


To clarify, this version tries to fix the synth rebuild-repository and synth prepare-system and similar commands. The flavor stuff really threw a wrench in the works on those.
Click to expand...

Tried it out on my system:

Code: 
# synth rebuild-repository
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Stand by, building pkg(8) first ... done!
Stand by, prescanning existing packages.
Removed: pkg-1.10.3.txz
Stand by, recursively scanning 139 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Note: This system's pkg(8) is configured with CONSERVATIVE_UPGRADE = true
      You may wish to toggle that setting if this is a local repository.

root@forklift:~ # synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt

root@forklift:~ # synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 139 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
I'm not seeing flavor-related errors anymore. I'll play with it some more, but it looks okay so far.
 
I'm not able to test this patch until it goes live.

The issue is happening on my production server, and I can't have the system installing all the Synth dependencies when the resources are needed elsewhere.
 
Mayhem30 said:
I'm not able to test this patch until it goes live.

The issue is happening on my production server, and I can't have the system installing all the Synth dependencies when the resources are needed elsewhere.
Click to expand...

I'm not sure what you mean. It takes the same amount of resources to build a release as a git tag. Unless you mean you have a wait a week for it to appear on FreeBSD official package servers ...
 
marino said:
I'm not sure what you mean. It takes the same amount of resources to build a release as a git tag. Unless you mean you have a wait a week for it to appear on FreeBSD official package servers ...
Click to expand...

When I don't use synth to install or upgrade a port, it has compile all the dependencies because they only exist in the repo - which is not used during make install. Then all of them are installed - whether or not they are "build only".

It was just using a fair amount of resources .. and a bit of work to clean everything up afterwards.
 
well, 2.01 synth can build 2.02 synth package, and you can install packages directly with pkg add and pkg remove commands. So you don't have to build outside a jail.
 
Mayhem30 said:
It's not too bad to rebuild an index on SSD's, but it sure takes a while on standard HDD. I realize that is beyond your control, but you'd think FreeBSD would have better tools in place so this wouldn't be necessary.

Is this how ports-mgmt/poudriere works as well to handle flavors? Building an index every time the port changes?
Click to expand...

well, it's not every time a port changes. It's every time the ports tree changes.
poudriere doesn't do this. The index is essentially a cache. Synth does it once, then looks up the data. Poudriere determines it every time. The additional scan time is repeated on a limited number of ports (as opposed to the whole tree) every time poudriere is launched. Synth is going to perform a lot fast during the actual building.

If freebsd actually provided an equivalent index with the tree, then it wouldn't be necessary.
There's no way of knowing how many potential packages there are without a full scan. Remember, there's no longer a 1:1 ratio between packages and ports. The only way to figure out how many packages per port there is is to do a VERY lengthy per port scan.

Incidently, see http://www.ravenports.com.
Ravenports does a full tree scan in less than 1 second.
Extrapolating to a tree the size of freebsd, I think ravenports could scan the tree in about 2, maybe 3 seconds.
Ravenports blows freebsd ports away in areas like these (and freebsd blows pkgsrc away in the same areas)
 
marino said:
Ravenports does a full tree scan in less than 1 second.
Extrapolating to a tree the size of freebsd, I think ravenports could scan the tree in about 2, maybe 3 seconds.
Ravenports blows freebsd ports away in areas like these (and freebsd blows pkgsrc away in the same areas)
Click to expand...

By the way, the ravenports scan is done serially (one port at a time) while the freebsd ports scan is done in parallel (e.g. 12 builders at a time on a 4-core system). Without the parallel scanning, the scan would take like 10 times longer (conversely if the ravenports scan was done in parallel, it would cut 90% off the < 1 second time).
 
This is what I'm seeing with the new version this morning.
Code: 
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.02
====================================================================
               Copyright (C) 2015-2017 John R. Marino

$ sudo synth status
Querying system about current package installations.
Installed package ignored, mail/py-authres package unmatched
Installed package ignored, dns/py-dns package unmatched
Installed package ignored, security/py-fail2ban package unmatched
Installed package ignored, devel/py-ipaddr package unmatched
Installed package ignored, mail/postfix-policyd-spf-python package unmatched
Installed package ignored, devel/py-pyinotify package unmatched
Installed package ignored, mail/py-pyspf package unmatched
Installed package ignored, devel/py-setuptools package unmatched
Installed package ignored, databases/py-sqlite3 package unmatched
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt


$ sudo synth prepare-system
Querying system about current package installations.
Installed package ignored, mail/py-authres package unmatched
Installed package ignored, dns/py-dns package unmatched
Installed package ignored, security/py-fail2ban package unmatched
Installed package ignored, devel/py-ipaddr package unmatched
Installed package ignored, mail/postfix-policyd-spf-python package unmatched
Installed package ignored, devel/py-pyinotify package unmatched
Installed package ignored, mail/py-pyspf package unmatched
Installed package ignored, devel/py-setuptools package unmatched
Installed package ignored, databases/py-sqlite3 package unmatched
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
 
in other words, the unmatched packages were installed from pre-flavors, so there are no flavor annotations provided with the internal "pkg query" commands that synth is issuing.
 
Thank you, removing / installing all the ports solved the issue.
Code: 
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt

$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
 
John marino, I'm having no trouble building flavored ports, for example; net/py-speedtest-cli@py36 builds all dependencies with python3.6 ports. However when I want to build a non flavored port like net/samba46 that depends on python against python3.6 it doesn't work?
I have this in LiveSystem-make.conf:
Code: 
DEFAULT_VERSIONS+=python=3.6
When I use synth status build.list
I get this:
Code: 
N => devel/py-iso8601@py27
N => dns/py-dnspython@py27
Which are dependencies of samba. Are the DEFAULT_VERSIONS entries ignored now that flavors are here?
 
doubtful. I suspect the py27 versions are specified. The "default" is used when no version is specified. However, a good deal of s/w doesn't build with python higher than 2.7, so for those, python 2.7 has to be required (and for those, the requires version of python ignored). Looks normal to me. You're going to have multiple versions of python modules installed.
 
So using my home 'play' server. I deleted all the packages that were previously in here:
Code: 
$ ls -lhG /var/synth/live_packages/All
total 0

...then....

Code: 
# synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.02
====================================================================
               Copyright (C) 2015-2017 John R. Marino


# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Installed package ignored, graphics/py-pillow package unmatched
Installed package ignored, missing from ports: devel/py27-setuptools
Installed package ignored, x11-toolkits/py-tkinter package unmatched
Stand by, comparing installed packages against the ports tree.
Stand by, building pkg(8) first ... done!
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the packages that would be fetched:

Removed / reinstalled 'prime' packages (see post from John).

Code: 
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the packages that would be fetched:
  => ccache-3.3.4_8.txz (devel/ccache)
  => indexinfo-0.3.1.txz (print/indexinfo)
  => gettext-runtime-0.19.8.1_1.txz (devel/gettext-runtime)
  => gettext-tools-0.19.8.1.txz (devel/gettext-tools)
  => pkgconf-1.3.10,1.txz (devel/pkgconf)
  => gmake-4.2.1_1.txz (devel/gmake)
<more>


# synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
The following packages will be fetched:

New packages to be FETCHED:
   ccache-3.3.4_8 (95 KiB: 0.00% of the 5 GiB to download)
   indexinfo-0.3.1 (6 KiB: 0.00% of the 5 GiB to download)
   gettext-runtime-0.19.8.1_1 (147 KiB: 0.00% of the 5 GiB to download)
   gettext-tools-0.19.8.1 (2 MiB: 0.04% of the 5 GiB to download)
   pkgconf-1.3.10,1 (51 KiB: 0.00% of the 5 GiB to download)
   gmake-4.2.1_1 (378 KiB: 0.01% of the 5 GiB to download)
<more>

..then eventually
The task is complete.  Final tally:
Initial queue size: 6
    packages built: 6
           ignored: 0
           skipped: 0
            failed: 0

Duration: 00:08:55
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 259 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt

The "regenerating flavor index" on my machine takes about 10 minutes.
 
ouch. There is low and moderate hanging fruit that could easily speed the scans up 5 fold (e.g. 2 minutes rather than 10) but so far port manager hasn't seen the current implementations as a problem. sorry; I can't fix ports here.
 
You must log in or register to reply here.
Back
Top