Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

Once more, and if you hate me:

Update for ftp/curl and www/youtube_dl.
Code: 
cmake-3.4.2.txz failed dependency check.
libquvi-0.4.1_4.txz failed dependency check.
ffmpeg-2.8.5,1.txz failed dependency check.
webkit-gtk2-2.4.9.txz failed dependency check.
raptor2-2.0.15_1.txz failed dependency check.
gstreamer1-libav-1.6.3.txz failed dependency check.
gimp-app-2.8.16_1,1.txz failed dependency check.
alsa-plugins-1.1.0.txz failed dependency check.
chromaprint-1.1.txz failed dependency check.
gstreamer1-plugins-chromaprint-1.6.3.txz failed dependency check.
gstreamer1-plugins-curl-1.6.3.txz failed dependency check.
foomatic-db-20150819.txz failed dependency check.
foomatic-db-engine-4.0.12,2.txz failed dependency check.
wx30-gtk2-3.0.2_4.txz failed dependency check.
openimageio-1.5.20_3.txz failed dependency check.
vorbis-tools-1.4.0_10,3.txz failed dependency check.
rasqal-0.9.33.txz failed dependency check.
gimp-gutenprint-5.2.10_3.txz failed dependency check.
gutenprint-foomatic-5.2.10_1.txz failed dependency check.
gstreamer1-plugins-all-1.4_3.txz failed dependency check.
freedink-dfarc-3.12.txz failed dependency check.
git-2.7.0.txz failed dependency check.
libcmis-0.5.0.txz failed dependency check.
libxul-38.6.0.txz failed dependency check.
opencv-2.4.9_7.txz failed dependency check.
py27-gimp-2.8.16.txz failed dependency check.
cmus-2.6.0_3.txz failed dependency check.
cdrtools-3.01.txz failed dependency check.
synergy-1.7.5.txz failed dependency check.
redland-1.0.17_4.txz failed dependency check.
php56-curl-5.6.17.txz failed dependency check.
gutenprint-5.2.10.txz failed dependency check.
mpv-0.15.0,1.txz failed dependency check.
phonon-gstreamer-4.8.2_1.txz failed dependency check.
vlc-2.2.1_6,4.txz failed dependency check.
transmission-cli-2.84_4.txz failed dependency check.
transmission-daemon-2.84_3.txz failed dependency check.
transmission-gtk-2.84_5.txz failed dependency check.
freeciv-2.5.2.txz failed dependency check.
freedink-108.4.txz failed dependency check.
git-review-1.24_2.txz failed dependency check.
osmo-0.2.14.txz failed dependency check.
firefox-44.0,1.txz failed dependency check.
midori-0.5.11.txz failed dependency check.
ap24-mod_security-2.9.0.txz failed dependency check.
nspluginwrapper-1.4.4_5.txz failed dependency check.
webkit-gtk3-2.4.9.txz failed dependency check.
clamav-0.99.txz failed dependency check.
blender-2.76b.txz failed dependency check.
feh-2.14.txz failed dependency check.
gimp-2.8.16,2.txz failed dependency check.
gstreamer1-plugins-opencv-1.6.3.txz failed dependency check.
mupdf-1.8,1.txz failed dependency check.
openshadinglanguage-1.6.6.txz failed dependency check.
virtualbox-ose-4.3.34_1.txz failed dependency check.
qt4-qtconfig-4.8.7.txz failed dependency check.
icedtea-web-1.5.2.txz failed dependency check.
thunderbird-38.5.0.txz failed dependency check.
codelite-9.0.txz failed dependency check.
libreoffice-5.0.4_1.txz failed dependency check.
cclive-0.7.16.txz failed dependency check.
gtk-youtube-viewer-3.2.0.txz failed dependency check.
redports-node-0.1.2.txz failed dependency check.
transmission-2.84.txz failed dependency check.
fvwm-crystal-3.0.6_8.txz failed dependency check
gimp-app, blender, thunderbird, virtualbox-ose, webkit-gtk3, ibreoffice !!! .... libreoffice alone is 2 hours, all will be about between four and five hours ... that is impossible to do.
How I wrote above three hours per day or a whole day in the week...

That's enough ...finished, forget it.
 
If you've given up: maybe it's a good thing. You seem to have some misunderstandings (or disagreements) how building should work and those conceptions seem to be a source of frustration for you. I would advise not rebuilding daily, because stuff like bumping curl will drive you crazy.
 
Did you not understand, that is impossible to do. The system is the half day busy with recompiling something. That is much too extreme. There must be an other way. I hope portmaster (I don't like It) will never deprecated.
If this the only way the concept of shared libraries failed.
 
Last edited by a moderator:
1) you are free to build a tool that works like you think it should
2) portmaster has same constraints. If you choose not to rebuild dependencies, you will break your system at some point, pretty much guaranteed.
3) Who said you have to build daily? Why not once a week, at night? Or once every 2 weeks, month? Your pain is self-inflicted.
4) it's quite possible. Have you noticed nobody else is agreeing with you? What does that say?

Thanks for giving it a try, and I'm sorry Synth is not working out for you. Portmaster's future is not bright and I recommend you at least use poudriere, but it acts like synth so you will probably hate it too.
 
Code: 
Have you noticed nobody else is agreeing with you?
Yes and I wonder about. I doubt everybody has the capacity to do this.

Ok, I try it once per week, but these will also bulk of packages and hours to compile.
 
do what, exactly?
1) you are on the ports head, not the quarterly. The head moves fast.
2) just about every day, somebody makes a commit the breaks hundreds if not thousands of ports. Libreoffice and KDE pretty much break every day
3) if you try to build every day, this is what you and everyone should expect

thus: don't build every day.

Remember what synth is doing: building an up-to-date repository for 1 or more systems to update themselves. By it's very nature, it's compiler intensive. You should be doing it periodically, not continuously.

Also note that you are not updating single packages, but the entire system. Maybe synth upgrade-system is not for you. Maybe you should identify specific packages you to update and just update those, and then stuff like curl won't bite you. In other words, you have a very demanding concept of operations, but you are unwilling to accept the consequences. You either need to change your operations or change your expections. (dropping synth is a way of changing both)
 
I think I've tried to explain this already many times but the crux of the matter is dynamic linking done by ld.so(1) and the way link library dependencies are so incredibly inflexible as done in almost every other UNIX/UNIX-like OS. We have no choice on the matter because FreeBSD doesn't provide its own methods/ways doing those, they were all taken as given from other systems where those features were developed and while alternatives do exist nobody seems to use them. Also the ELF file format that is behind all this is taken for granted as the standard file format, everyone expects that ld.so(1) works more or less the same on all UNIX/UNIX-like OSes.

In a nutshell you have port A that provides a shared library libfoo.so.1. That library file exports a set of symbols that makes the API/ABI of the module libfoo and its version this API is 1 as noted by the suffix. Now there's port B with an executable program named foobar that needs the library libfoo.so.1 so naturally it gets it from port A, B now depends on A. What is done with this program foobar is that at the linking stage of the build of the executable a reference is inserted to the shared library it depends on so that the dynamic linker will know to load the shared library, libfoobar.so.1, also references to the unresolved symbols to this shared library are left in the executable and those are resolved by the dynamic linker when the program is loaded.

The tricky parts is that if there is ever an update to the port A that causes the API/ABI provided by libfoo.so to change in such a way that the version number has to be bumped up to 2 (making the library libfoo.so.2) it causes a problem for the program foobar in port B because it's now hardwired to load libfoo.so.1 by the dynamic linker at startup. So what is the solution you ask? Of course rebuilding of the executable foobar with the updated port B installed with the new library libfoo.so.2 so that the correct library is referenced and more importantly the executable foobar is made to use the correct API/ABI because of the change in the dynamic link library. Now the real kicker, this solution (rebuild of the dependent executable) is the only solution in existence if using direct dynamic linking with ld.so(1) or dlopen(3)/dlsym(3) methods. Now think about what this means when you have a port that is used by a huge number of other ports such as the mentioned ftp/curl and then the dependents of the ports that depend on ftp/curl.

There are other ways of dealing with the dependencies and API/ABI concerns that are very much superior from the application's and the developer's point of view, the most widely used is the COM/COM+ as you find it in MS Windows:

https://en.wikipedia.org/wiki/Component_Object_Model#COM.2B_and_DCOM


The downside of those methods is that they introduce extra baggage that is fine on user application level but may not be desirable on the system service/utility level.
 
I just pushed version 0.99_3. Note a major change of action in the synth build-repository (it was renamed and new command took over the name). The removal of "Escape" for graceful shutdown was a big change too.

Code: 
build/synth: bug fixes, new features

WARNING: rebuild-repository command has changed action!  see below!

The follow changes have been made since v0.99_2:
  * Change the graceful shutdown key from "Escape" to Control-C.
    The former was easy to hit inadvertently (reported) and could be
    interfered with by terminal ANSI codes and/or mouse wheels.  The
    documentation has also been modified to reflect this change.
  * Fixed bug where installed packages that no longer had a port
    might cause the scan to fail rather than be ignored as advertised
  * New feature: SYNTHPROFILE environment variable
    When SYNTHPROFILE is set toTill be loaded rather than the default
    profile.  This is aimed for synth's use in scripts.
  * The "rebuild-repository" command has been renamed to "prepare-system".
    This is partly because the former command will be repurposed.
  * A new command assumed the name "rebuild-repository"; it performs a
    sanity check on all the built packages, removes the bad ones, and
    rebuilds the local Synth repository on command.  It is primarily for
    scripting use, but it has other legitimate uses.
  * Fix case where prefetching packages would try to update a non-existent
    local Synth directory.  As a consequence, prefetching is only done
    from a single external repository (the normal use case thought)
 
Oops, there's a bad typo in the commit message. The new shutdown sequence is Control-Q, not Control-C !
 
marino@ said:
Oops, there's a bad typo in the commit message. The new shutdown sequence is Control-Q, not Control-C !
Click to expand...

Seen that and was about to post a message about the parade of disgruntled synth users coming shortly to post it's broken. :)

After using Synth for the last couple of weeks now, I'm really liking it. Much easier to setup than Poudriere and seems subjectively faster as well.
 
There's an earlier post saying a specific job takes poudriere 24 hours but Synth does the same job in 18 hours (on the given machine). So it's more than "subjectively" faster. :)
 
I have 900+ ports installed on my workstation. Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository. I don't know how long Poudriere took on the same machine but I know it was longer. I thought there was a mistake and ports were not getting rebuilt, but after checking, they were all in fact rebuilt. It is quite fast.

Again, I appreciate the tool and work your putting into it. Thanks!
 
I am also using Synth from the first day and after all this years which I "spent" with portmaster I am so happy and thankful that is Synth in the ports and for Marinos help too.
Thank you.
BTW: I start synth-upgrade system before bad time and in the morning is everything done :).
 
protocelt said:
Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository.
...............
Again, I appreciate the tool and work your putting into it. Thanks!
Click to expand...

I need to understand devel/ccache and its requirements. Memory, CPU, etc. I'd love to use this. Feel free to PM me, anyone!

Yes its a great tool; I can't wait to roll it out to all my machines.
 
protocelt said:
I have 900+ ports installed on my workstation. Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository. I don't know how long Poudriere took on the same machine but I know it was longer. I thought there was a mistake and ports were not getting rebuilt, but after checking, they were all in fact rebuilt. It is quite fast.

Again, I appreciate the tool and work your putting into it. Thanks!
Click to expand...

Keep in mind that ports-mgmt/poudriere goes to very great lengths to ensure an absolutely clean build environment and that includes providing a pristine base system for every single individual build, that is implemented by jail cloning which is quite slow on UFS and on ZFS is still a minor slowdown. Synth on the other hand makes use of the running live system by nullfs(5) mounting the needed directories over the chroot(8) environment which is very fast.
 
I would argue that synth chroots are just as clean and pristine as poudriere's.
The only thing the jails buy you over chroot are:
1) Jails can kill rogue processes efficiently where those may cause issues for synth (inability to dismount, etc). I tried to implement watchdog to address that but it's only 99% effective. that's still a work in progress
2) Jails can turn off network per phase (desirable for it's purpose) but I don't care about that

Other that #1, there's not much advantage to the jail over chroot for a typical user.
 
I pushed a relative minor update to synth today:

Code: 
ports-mgmt/synth: new feature, minor bug fixes and improvements

This is a minor update to synth, which includes:
  * Support for the WHYFAIL environment variable.  If this variable
    is defined (to any value) in the environment, Synth will turn on
    the "debug" mode for dependency and option sanity checks.  This
    mode will provide exact details on how the package failed the check.
  * README.md: editorial corrections, 3 images replaced to reflect current
    version of Synth
  * Man page: editorial correction, WHYFAIL documented, and the "Impulse"
    indicator was documented (in NOTES section)
  * Significantly improve ports scan error messages.  In particular,
    eliminate the 'bad value ""' messages that are caused by ports that
    are partially or completely missing.  Also propagate exception
    messages when helping.
  * Log 03 (ignored ports) did not list the actual ports, only the reason
    the port was ignored.  Fix bug to show category/port too.

Erratum on previous commit message: The "Graceful Shutdown" is initiated
with Control-Q, not Control-C!  The typo is doubly unfortunate because
Control-C will exit Synth without cleaning up the mounts.
 
One question more, please:
I have all packages built and rebuilt to the last version. Is it good idea to reinstall everything?
Thank you.
 
fernandel said:
One question more, please:
I have all packages built and rebuilt to the last version. Is it good idea to re install everything?
Thank you.
Click to expand...

It's not really needed but # pkg upgrade -f is safe to run (if it wasn't we'd have big problems) and usually doesn't take too long. It's needed when you switch from one repository to another or there is a major change in the repository you're using such as a change in the default version of perl.
 
I don't think that is necessary. On the other hand, it doesn't hurt anything to do that either.
I do this "bulletproof" update technique form time to time to remove unwanted packages. You could the same perhaps:
https://www.dragonflybsd.org/docs/howtos/HowToDPorts/#index4h1

basically you'd remove everything and reinstall the key packages from Synth repository and it would pull in the dependencies as a result, so there would be no extra packages installed at the end.
 
I pushed 0.99_5 to fix a minor regression that nobody told me about. :)

Code: 
ports-mgmt/synth: Fix end-run regression

A fairly recent change caused a regression after a build was complete.
Previously a "tally" or summary of the build would appear after the
ncurses screen was restored to the regular terminal mode.  It would
list how many ports were built, failed, etc.  After the regressin, it
just ended abruptly.

This commit restores the tally to show as it did previously.
 
Minor Feature Request:
aaaa deeefffaaaulllt niiicceee vvaaaallluuuee please ;)
 
You must log in or register to reply here.
Back
Top