Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

PacketMan I wouldn't recommend Quarterly until the next version comes out in April. I think the ports bugs that synth finds won't generally be fixed in Quarterly.
 
(but you can of course petition for those fixes to get backported and that should happen in general)
 
PacketMan said:
...synth is continuing having to build a pile of ports..... Thoughts?
Click to expand...

This is a good thing, in my view a very good thing and characteristic of continuous integration of such a huge collection of disparate sources. As marino points out the totality of sources required by a big program, e.g. webkit2-gtk3, change every day, sometimes every hour! FreeBSD is unique amongst the linuxes and bsd's in continuously integrating the operating system and ports. There are FreeBSD build servers continuously building each branch of the system such as 10-STABLE, and the head of the operating system, 11-CURRENT. There are also FreeBSD build servers continuously taking those latest builds of the operating system and building the latest releases of all 28,000 ports, and also the latest quarterly stable ports. On a FreeBSD system freebsd-update and pkg upgrade can pull from that immense body of work and quickly update local machines.

You can see the status of the latest builds on the FreeBSD package build server at Portsmon Check for all sorts of port changes and info at Freshports.

If you want to conduct your own local continuous integration you at least want to replicate the quality of FreeBSD in maintaining self-consistent repositories -- everything in your local repository should be built against the other things that are present. You don't want the repository to contain a package that was built on top of something that you've changed in the meantime. It has been proved by a lot of experience (e.g. with Arch Linux) that that inevitably leads to mysterious failures. Programs lock up, crash, behave bizarrely and eventually someone figures out that you "just need to rebuild X". The only trustworthy path of development is to build self-consistent repositories in the first place.

Consider what self-consistent building means by looking at the list of libraries used by webkit2-gkk3:
Code: 
Shared Libs required:
   libgtk-x11-2.0.so.0
   libxslt.so.1
   libsoup-2.4.so.1
   libpango-1.0.so.0
   libjpeg.so.8
   libicuuc.so.55
   libsqlite3.so.0
   libgstapp-1.0.so.0
   libcairo.so.2
   libgtk-3.so.0
   libxml2.so.2
   libwebp.so.5
   libfreetype.so.6
   libcairo-gobject.so.2
   libsecret-1.so.0
   libatk-1.0.so.0
   libgstfft-1.0.so.0
   libgstreamer-1.0.so.0
   libicui18n.so.55
   libharfbuzz-icu.so.0
   libgmodule-2.0.so.0
   libgdk_pixbuf-2.0.so.0
   libenchant.so.1
   libgio-2.0.so.0
   libXcomposite.so.1
   libXt.so.6
   libpangoft2-1.0.so.0
   libglib-2.0.so.0
   libgobject-2.0.so.0
   libXrender.so.1
   libGL.so.1
   libharfbuzz.so.0
   libEGL.so.1
   libX11.so.6
   libgstpbutils-1.0.so.0
   libXdamage.so.1
   libgstvideo-1.0.so.0
   libgsttag-1.0.so.0
   libgdk-x11-2.0.so.0
   libpng16.so.16
   libgstaudio-1.0.so.0
   libintl.so.8
   libfontconfig.so.1
   libgstbase-1.0.so.0
   libgnutls.so.28
   libgdk-3.so.0
   libpangocairo-1.0.so.0
If any of those libraries is changed webkit2-gtk3 should be rebuilt.

Now pkg doesn't always re-install every package that a port depends on, it tries to figure out from library sonames if that is necessary. But sometimes programmers make significant changes in a library and neglect to renumber the library to reflect the "abi change". Maybe the library itself is ok but some ancilary script changes. It isn't always clear when it's necessary to force rebuilds. We're dealing with a huge pile of complicated interdependencies created by tens of thousands of programmers some of whom are unqualified hacks and the result is what I call not "layers" of software but a "Pile of Software" or sometimes a "Pile of Shit". So in the (correct) FreeBSD approach of continous integration even if pkg doesn't re-install everything, everything is in the repository -- correctly built against everything else. On FreeBSD if you do get into bizarre failures after updates you can get a completely consistent instatllation, whether from FreeBSD binary packages or from your local synth repository, by executing pkg upgrade -f.

tldr: The goal of synth or poudriere is to build a complete, self-consistent repository. It's a challenging goal. As everyone seems to be discovering in this thread synth does an especially good job of it.
 
garry said:
if any of those libraries is changed webkit2-gtk3 should be rebuilt............dealing with a huge pile of complicated interdependencies......
Click to expand...

If rebuilding ports to ensure they all link with all the other ports is so darn necessary because we dealing with a huge pile of complicated interdependencies why are prebuilt packages even available for downloading? I've got a linux machine that only takes minutes to update, not 50 something hours just because 6 ports actually need to be upgraded! I'm not sure I am further ahead with synth. At least with portmaster what was needed was what got rebuilt. On a GNOME 3 machine synth is basically saying lets rebuild 300+ ports only to have 6 or so ports actually upgraded.

Over the next few days I will likely start doing upgrades months apart to (a) hopefully take advantage of pre-built packages, and (b) simply have less 'downtime'. That just said if I can get one 'master' machine doing all the heavy lifting (rebuilding all the ports that are in use over all my machines) and then use Bittorrent Sync to distribute the packages to the other machines, and then issue one single command synth upgrade-system to simply install those packages that were built by the master, with near zero ports being rebuilt then I will likely continue using synth. If synth says yeah sorry still gonna rebuild those ports on these smaller machines because something is not pristine perfect then I would say I will likely stop using synth and go back to portmaster.

The best technical solution is not necessarily the best practical solution.
 
PacketMan@ you don't need synth on the other machines, only pkg(8).
You're not getting what a repository is, otherwise you wouldn't have even mentioned running synth on the smaller machines.


Going back to portmaster because it doesn't make you do stuff is like changing doctors to one that won't tell you that you have cancer.

The linux comparison is terrible. There are no options, so there's no reason to build from source. The equivalent is using stock FreeBSD packages which take exactly the same time as the linux case.

Also, why don't you take a look at the recent gnome3 thread in this forum and see what a smashing job portmaster did with it.

Plus, what is this "downtime" you speak of?
synth packages are not installed as you go (unlike portmaster). Even with "synth prepare-system" nothing is upgraded. When pkg(8) finally updates the system it takes seconds or at most a few minutes for hundreds of packages.

Nowhere above are you comparing apples to apples.
 
The downtime is simply this: a machine is pretty much unusable when the cpu is 100% doing all this rebuilding.

Doctors are a good comparison considering how often they get you to do stuff that is unnecessary. Which I why fired mine and got a new one that simply instead helped me work with a certified asthma specialist and now I enjoy breathing again.

Referencing linux: since my machines are 99% port stock (and I can make them 100% stock if I want to) again why is all the rebuilding necessary?

You are right, I do not (fully) understand what a repository is, I'm a newbie, but it was you that previously said I could/should point synth on the other machines to one single repository on one machine. I now (think I) know I can copy the packages to the other machines and use pkg to install them; no synth at all. But again I think that supports what I am saying? -- why do we have to do all this rebuilding to guarantee all these complicated interdependencies are correct (and when most of that output is not even used) when I can simply bypass all of that by just installing packages that were prebuilt by another machine?

I have not had any issues with my GNOME3 machine and using portmaster to update it. I read the usr/ports/UPDATING and plow through it. Takes a restart or two to fix an issue I missed in a new update, but at least I am compiling only 30 ports and not 300+.

Thanks for the feedback Marino, I am gonna carry on here and I will report back in a few days or in a week or two. (Depends on how much snow I get....got a nice snowmobile just purring to get out.)
 
PacketMan said:
If rebuilding ports to ensure they all link with all the other ports is so darn necessary because we dealing with a huge pile of complicated interdependencies why are prebuilt packages even available for downloading? .....

If synth says yeah sorry still gonna rebuild those ports on these smaller machines because something is not pristine perfect then I would say I will likely stop using synth and go back to portmaster.
Click to expand...

The FreeBSD prebuilt packages are built correctly. Their build server has done all of the work I described. Those build servers are VERY busy machines, doing all that work so that we don't have to. They use poudriere and they rebuild packages greedily. The difference in behavior between FreeBSD and say Arch Linux (which is also a rolling system) is this:

In Arch Linux you can update your binary packages every day with pacman -Syu much as we do pkg upgrade in FreeBSD. The Arch packages are built "lazily" with each developer throwing updated binaries into the pile as he sees the need to upgrade to a newer version or rebuild to accomodate changes in dependencies. Because most dependencies are through dynamic libraries and in the perfect case they can be updated independently of their dependents stuff mostly updates day-by-day with relatively few packages being downloaded. But because of the unconstrained chaos of complexity and undisciplined programming that I described it also often happens that an update does break something. Then users are expected to file bug reports, the developers track down the problem, maybe find that "Oh! Y will need to be rebuilt now that we've updated X" and they bump the revision on Y, rebuild it and push out a new e.g. webkit2-gtk3 package and everyone is happy, after having a broken system for a day or two.

The same scenario happens with portmaster. Search these forums to find numerous instances of people breaking some software by doing lazy upgrades with portmaster.

With FreeBSD binary updates it is also possible for your local machine to end up with a broken epiphany web browser because a cairo update broke webkit2-gtk3 BUT you can fix the problem instantly by re-installing your packages -- they are being drawn from a repository that itself is never broken in this way. It is always clean and self-consistent.

You're not wrong of course. The whole repository-building approach requires a lot of duplication of builds and is frustrating. You may be happier with portmaster. We're dealing with the problem with how to put together a reliable system from a large number of independently developed sources and doing it properly IS AN UNSOLVED PROBLEM. Nix is an attempt to solve the problem from a better foundation. They may have cracked the problem of reliable atomic upgrades! You can learn a lot about The Problem starting from Eelco's PhD thesis.
 
1. That's not the definition of downtime.
2. If you have or want 100% stock then don't build at all. Use packages.
3. Who said you have to build daily? Can't you build weekly or monthly?
4. You're on the bleeding edge ports. You have to expect this. If you don't want change, get static branches.
5. The primary guys that wrote pkg, poudriere, and synth and all the experienced ports developers all say there's only one correct way to build and they all say portmaster isn't doing it correctly. So we have a stack of resumes on one side yet the portmaster defenders have nothing like that except "it works for me". Again, take a look at the gnome3 removal thread to debunk that. Surely you don't think that *ALL* of us are wrong but guys with zero credibility are right?
6. I have no idea what you are asking about the respository. We told you to have one common repository, you said that wasn't "practical" and you wanted to copy it. Fine. What does that have to do with rebuilding everywhere? You have one machine for building, the remaining machines consume the packages. You only need one copy of synth to produce the repository.
7. 300 ports is NOTHING. My home machine can hit 1000 ports an hour, our blade machine can hit 2800 ports per hour. If you use ccache, it will rebuild huge ports in minutes. And again, how often is this happening? A few hours every 2 weeks?

Maybe you just need to adjust you concept of operations. (use stable, don't build daily, etc, etc)
 
garry said:
You may be happier with portupgrade.
Click to expand...

Thanks Gary, I think I will re-read your last 2 posts a few times....is that a dim light I see coming on in the back of my head. :oops:

No - I don't use portupgrade, I use portmaster on a couple machines (which I thought was different than portupgrade), and synth on others. I will be happier when I buy a Dodge Hemi 440 powered 92 core 82000Ghz liquid nitrogen cooled processor! :D Actually eyeing some sales on a 6-core cpu and motherboard.
 
marino@ said:
1. That's not the definition of downtime.
2. If you have or want 100% stock then don't build at all. Use packages.
3. Who said you have to build daily? Can't you build weekly or monthly?
4. You're on the bleeding edge ports. You have to expect this. If you don't want change, get static branches.
5. The primary guys that wrote pkg, poudriere, and synth and all the experienced ports developers all say there's only one correct way to build and they all say portmaster isn't doing it correctly. So we have a stack of resumes on one side yet the portmaster defenders have nothing like that except "it works for me". Again, take a look at the gnome3 removal thread to debunk that. Surely you don't think that *ALL* of us are wrong but guys with zero credibility are right?
6. I have no idea what you are asking about the respository. We told you to have one common repository, you said that wasn't "practical" and you wanted to copy it. Fine. What does that have to do with rebuilding everywhere? You have one machine for building, the remaining machines consume the packages. You only need one copy of synth to produce the repository.
7. 300 ports is NOTHING. My home machine can hit 1000 ports an hour, our blade machine can hit 2800 ports per hour. If you use ccache, it will rebuild huge ports in minutes. And again, how often is this happening? A few hours every 2 weeks?

Maybe you just need to adjust you concept of operations. (use stable, don't build daily, etc, etc)
Click to expand...

With all due respect I take offence to this post. An OS and its tools successfully making it in the market accommodates a variety of thinking and user groups. I may have zero creditability but I am telling you I have had zero issues maintaining a GNOME3 machine with portmaster, and I don't have selective memory thank you. I am sorry I don't have the resume you have, or a 28000 port per hour blade machine either. Maybe your release notes should say something like "synth is only for people with fancy resumes and big budget servers so they can tolerate rebuilding 300+ ports EACH AND EVERY TIME only 6 or so ports actually need to be upgraded. No one said I had to build daily thank you, I was doing it to provide valuable feedback to you, but since you are clearly right and I am clearly wrong (and apparently in a lesser class too) I will take a break in reporting my findings. Oh, I am using ports because guys in this forum who I put a lot of weight and trust in their experience advised me to stick with ports.

Bye! ........Come on snow.
 
I'm not sure what you are taking offense at.
Either you believe the logic is correct or you don't. If you don't, and your belief against what all the experts are saying, I'm curious how that's reconciled.
You didn't respond to any of the implied questions, the primary ones being: How often are you rebuilding these things?

and why are you so sure other methods would have upgraded those 6 ports?

The "each and every time" line is a crock though. If you don't portsnap/svn update the ports tree, it won't rebuild at all. It's rebuilding because you are changing the tree. As all the stuff about beefy machines and blades (missed the point). I'm saying 300 ports is nothing, even on wimpy machines, especially when the machines use ccache. But 300 ports per day every 2 weeks is only 600 ports a month. Even the freebsd servers are only building once every 7 days.
 
I got an interesting request to make Synth detect cached options files that are identical to the default options and delete them. Synth can't do that because the options are mounted read-only, but I came up with a good alternative. I added the script directly to ports. Here is the commit message:

Code: 
Add new tool script: redundant-opt-files.sh

I got a request to make Synth identify "redundant" cached option files,
where "redundant" means the saved port options are identical to the
default options.  For Synth (and portmaster?) which use the port's
cache options, these redundant files are somewhat of a liability.  At
best they do nothing (Synth assumes default options) and at worst they
will cause a future build to stop if the maintainer changes the port
options later.

This situation is avoidable.  Rather than build detection into Synth,
I decided to write a generic shell script for ports.  When run, it
will display the full path to the port's options directory if the
cached options are the same as the defaults.  This output is suitable
to pipe to "xargs rm -rf" to remove all the redundant options in a
single command.

https://svnweb.freebsd.org/ports?view=revision&revision=408849

Once somebody decides to switch completely to Synth, it's a good idea to run this script.
In fact, it's a good idea to run the script *before* you run Synth the first time.
I found 122 identical cached option files on my machine.
 
Doh! I wish that script was there a few days ago before I went through every port in that directory manually! Yeah I think I deleted well over 100.
 
I decided to try devel/ccache. I just made a directory in /var/cache/ccache and I have as I read in /etc/make.conf:
Code: 
WITH_CCACHE_BUILD=yes
Is this enough or I need to do something more, please?
 
fernanel@ no, that's not correct. Remove that line from /etc/make.conf (if you put it there for synth).

All you need to do is ccache -s and copy the value of "cache directory" to synth configure screen.
Make sure you actually set up ccache first, see the man page (or ccache help screen) to size it correct and define the location if you don't like the default.
 
Some other weirdness that I'm trying to work out. I've seen pkg want to install math/gmp and lang/lua52 for no apparent reason quite a lot on apparently unrelated packages. For example:

Code: 
# pkg install devel/gmake
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
Synth repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gmake: 4.1_2 [Synth]
  gmp: 5.1.3_3 [Synth]
  lua52: 5.2.4 [Synth]

If I just use synth to do this:

Code: 
# synth install devel/gmake
Stand by, updating external repository catalogs ... done.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Scanning entire ports tree.
Scanning existing packages.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  66 KiB  67.6kB/s  00:01
Processing entries: 100%
Synth repository update completed. 220 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gmake: 4.1_2 [Synth]

Here is another example:

Code: 
# pkg upgrade -f dns/ldns
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
Synth repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  lua52: 5.2.4 [Synth]
  gmp: 5.1.3_3 [Synth]

Installed packages to be REINSTALLED:
  ldns-1.6.17_5 [Synth]

I'm wondering why there is different behavior between synth and pkg, but also why math/gmp and lang/lua52 are related at all to gmake or ldns in the first place as they are not dependancies of either?

Thinking my pkg database might be screwed in some way I actually tried doing pkg prime-list > list, pkg delete -ay, rm /var/db/pkg/local.sqlite, and pkg install `cat list` but this hasn't changed anything.
 
i don't know. Isn't that implying that synth is doing the correct thing and it's pkg that is acting weirdly?
 
Yeah, sorry, I know it's not actually a synth issue, more a pkg one. Actually.... Just tried disabling the FreeBSD repository again and only leaving the synth one. Then it only installs gmake. So I guess it's the FreeBSD repo being consulted and having different build options or something..... Strange though, because both devel/gmake and dns/ldns do not have any options files in /var/db/ports.
 
Is that a build- or runtime dependency? pkg will not install build dependencies whereas synth will have to build them.
 
Nope. That setting doesn't seem to make any difference. Only setting enabled to no for the FreeBSD repo stops it.

I have worked around it though, it appears I can run pkg install -r Synth gmake. It works then!
 
You must log in or register to reply here.
Back
Top