Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

protocelt

Daemon

Reaction score: 413
Messages: 1,253

lib32 just wasn't implemented. The linux support is brand new but I forgot about lib32. DragonFly doesn't have these. I imagine it's not difficult to implement (e.g. one additional if on freebsd)
Ok, thanks.

I ran into a potential issue with java/openjdk8 myself (socket timeout trying to connect to VM) so I'd be interested to hear if anyone has trouble with that one. I suspect the builders minimal /etc/services is missing a port/protocol there.
FWIW, I just finished building java/openjdk8 on CURRENT with synth(1) and had no obvious issues on my machine, at least not that I noticed thus far. It's still got 400+ ports to go however.
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

I don't understand what you mean.
If you don't make config or you issue make rmconfig then synth status gives you the standard dependencies (again assuming there are zero packages already built). Which is not a good assumption. "status" always give the incremental list of built ports, not the absolute.

I you want the absolute list, then you're back to the standard make all-depends-list ports command.
 
Last edited by a moderator:

fernandel

Aspiring Daemon

Reaction score: 220
Messages: 891

I don't understand what you mean.
If you don't make config or you issue make rmconfig then synth status gives you the standard dependencies (again assuming there are zero packages already built). Which is not a good assumption. "status" always give the incremental list of built ports, not the absolute.

I you want the absolute list, then you're back to the standard make all-depends-list ports command.
I was not clear...
When I run for example portmaster multimedia/ffmpeg it will first open options (as you wrote "2.Need ports with options other than defaults.").
 
Last edited by a moderator:

Beastie7

Well-Known Member

Reaction score: 177
Messages: 425

Just by glancing over it seems a lot more straight-forward and intuitive than poudriere/portmaster. Will definitely try this out. Good stuff.

edit: quick dumb question. Why the name Synth?
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

I was not clear...
When I run for example portmaster multimedia/ffmpeg it will first open options (as you wrote "2.Need ports with options other than defaults.").
Synth doesn't do any of that. It assumes that you already went to the port and redefined options (which caches them), otherwise it assumes the default options. It does not handle the options dialog at all. This is intentional.

What it will do is flag a port with cached options that are out of date. If it's requested to build that port, it will not start the build but will list the port(s) which bad options and tell the user to either make config or make rmconfig that port and try again.
 
Last edited by a moderator:
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

garry

Member

Reaction score: 62
Messages: 72

Synth is written in Ada.
gcc*-aux ports (gcc-aux, gcc5-aux, gcc6-aux) are versions of GCC that contain the Ada front-end. It's needed to compile the program (build dependency only)
In fact the synth code is so beautiful it got me interested in learning ada and I ordered a couple of the Ada '95 books. Thanks for synth, and for showing us a good example with your Ada code. Synth was easy to configure with the interactive synth configure. It was easy to tell it to use my non-standard location of distfiles, to use ccache, etc. I like the use of "profiles" and the per-profile make.conf. It chose good values for the number of builders and jobs and really loads my cpus well -- it was fast in building my repository. I like the continuously updating status displayed during the build. Showing the "lines" of progress into the build for each package as it's building is helpful.

I've been using poudriere(8) for building a local repository of about 3000 packages and may now be switching to synth(1).
 
Last edited by a moderator:

garry

Member

Reaction score: 62
Messages: 72

short, unique, memorable, pronounceable (/me looks at poudriere) and is the root of "synthesis", which the first definition probably most apt: or "synthesize" which is a nearly a synonym for "make" only mostly for chemistry
After looking "synth" over I thought the command should have been named "synthesize" and the "upgrade-system" subcommand could have been simply "system" so that the commonest command could have been synthesize system.
 

fernandel

Aspiring Daemon

Reaction score: 220
Messages: 891

Synth doesn't do any of that. It assumes that you already went to the port and redefined options (which caches them), otherwise it assumes the default options. It does not handle the options dialog at all. This is intentional.

What it will do is flag a port with cached options that are out of date. If it's requested to build that port, it will not start the build but will list the port(s) which bad options and tell the user to either make config or make rmconfig that port and try again.
Thank you. I gave it a try and it works very fast and I didn't get any errors yet :).
 
Last edited by a moderator:
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

One question, is there a reason synth(1) doesn't nullfs(5)? mount the /usr/lib32 directory as well? At least on my machines running 11-CURRENT and 10.2-STABLE this is the case. Ports like emulators/virtualbox-ose will bail on building due to requiring access to the lib32 base libraries.
I think I've implemented support for this if you want to try it out.

if so, change line 24 of Makefile from
Code:
GH_TAGNAME=    v1.3:bundle c83a9d9
to
Code:
GH_TAGNAME=    v1.3:bundle b21d685
then make makesum
then the standard make deinstall ; make clean ; make install command.
 

Beastie7

Well-Known Member

Reaction score: 177
Messages: 425

The use of a concurrent primitive language is really interesting to me which caused me to do some research into languages like it (ie, Erlang, Elixir, etc). I'd wager for something like a build system where you're handling many heavy tasks at once and in a world of multiple cores, the synchronous message passing helps with speed. At least to my understanding. Ada looks to be very readable also.
 

protocelt

Daemon

Reaction score: 413
Messages: 1,253

I think I've implemented support for this if you want to try it out.
Just tested it out. Seems to work as suggested. Thanks!

On a tangential note, is there any future possibility for adding some flags(overrides) for the build commands such as concurrent builders and max jobs? Certainly not a necessity but would be a convenience for a user that wanted to change things on the fly depending on how a system is being used at that time. Having said that, I do understand the aim was to be simple so just asking mostly out of curiosity.
 

Beastie7

Well-Known Member

Reaction score: 177
Messages: 425

Just tested it out. Seems to work as suggested. Thanks!

On a tangential note, is there any future possibility for adding some flags(overrides) for the build commands such as concurrent builders and max jobs? Certainly not a necessity but would be a convenience for a user that wanted to change things on the fly depending on how a system is being used at that time. Having said that, I do understand the aim was to be simple so just asking mostly out of curiosity.
This.
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

I'm not planning on doing that. It's a slippery slope that will result in an invocation mess as you can see with portmaster man page. You could always create another profile or two so it's simply a matter of switching profiles (and no, I haven't throught about selecting profiles via command either although one switch would be preferable to several)
 

protocelt

Daemon

Reaction score: 413
Messages: 1,253

I'm not planning on doing that. It's a slippery slope that will result in an invocation mess as you can see with portmaster man page. You could always create another profile or two so it's simply a matter of switching profiles (and no, I haven't throught about selecting profiles via command either although one switch would be preferable to several)
Understandable. I haven't tested the profile feature(s) yet so it wasn't on my mind when I wrote the above reply. I would think profiles would suffice just fine. Thanks
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

I've just updated synth in ports. From the log:
Code:
  The following changes have been implemented:
    * The builder command executer had been upgraded with a watchdog.  It
      does not monitor the overall time of a phase (things like fetch /
      checksum vary depending on the internet connection and the volume it
      needs to download ranges from bytes to gigabytes), but it does monitor
      log progress.  Each phase has a maximum amount of time allowed for the
      log to be static (measured in lines, not file size).  If the log is
      static for too long for that phase, the processes of the builder will
      be killed, and the builder log updated accordingly.
    * The load indicator was stuck at 0.00 for some named locales (only on
      FreeBSD) so this was resolved.
    * Ports tree scanning time was cut nearly in half by caching make
      variables on each builders make.conf
    * Support for /usr/lib/lib32 was added for FreeBSD
    * purge-distfiles command was improved by handling potential exceptions
      and fixing the case of 100-1023 Mb purged (range was too narrow)
    * Typos on man page fixed
    * The directory ${PORTSDIR}/packages are now ignored.  This is the
      default package location and any existing packages were getting
      treated as port directories.
    * Skip some additional questions/actions if a graceful shutdown was
      previously detected
Some of those are big items, so I think most people would be interested.
I'm not moving to 0.99 until the functionality to use prebuilt binaries for dependencies is implemented.
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

hmm, there may be a regression on ports using scons. I'm not sure what it is, scons just freezes as soon as it's invoked with no error message. Maybe something with the watchdog setting the process group? unclear.
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

When I use the forking version of synthexec, scons works again, so I think that's what I'll have to do. Apparently scons doesn't like to be the process leader.
 
OP
marino

marino

Daemon

Reaction score: 441
Messages: 1,091

I could not find a solution to the scons problem, so I've disabled the watchdog until future notice. As far as I know it's a bug with scons, but there are tool many ports dependent on scons. It may take a while to get resolved and re-enabled. I pushed 0.98_4.
 

kpa

Beastie's Twin

Reaction score: 1,818
Messages: 6,318

It seems to work fine but I'm wondering why I keep getting a "Unfortunately, the system upgraded failed" -message (it should say "system upgrade") everytime on # synth upgrade-system even if everything seems to work? What's the condition that is tested for failure? I have no other repositories enabled but the one created by synth. System is:

Code:
FreeBSD freebsd10 10.2-RELEASE-p10 FreeBSD 10.2-RELEASE-p10 #13 r294085: Fri Jan 15 16:35:16 EET 2016     root@freebsd10:/usr/obj/usr/src/sys/GENERIC  amd64
 
Top