Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

marino

marino

I've been working on new software for the past few weeks. It's called "Synth" and one of its main purposes is to replace portmaster/portupgrade. It it similar to poudriere in that it supports concurrent building and local respository creation, but it's aimed at the regular user. It is a drop-in tool, and there is no "jail building" or "ports tree" installing or anything like that. It uses the features of the system it builds on by default, but it builds everything in a clean environment and it builds exactly what is needed.

It's particular aimed at users that:
  1. Insist on building ports from source no matter what.
  2. Need ports with options other than defaults.
  3. Want system upgrades to be painless.
  4. Have multiple systems that they want to share a common, customized
    package repository.
  5. Think postmaster is great.
  6. Want to gain concurrent package building locally.
  7. Don't like package surprises.
  8. Don't understand or want to deal with poudriere.
I've written a "HowTo" with graphics that is published on github: https://github.com/jrmarino/synth (scroll down to README.md). There is also a man page, but it's a drier version of the HowTo.

I pushed synth to FreeBSD ports last night (as ports-mgmt/synth). That means it will not be available as a pre-built package for a few days.

For those that don't want to wait, they can build it themselves pretty easily.
  1. Install the dependencies: pkg install gcc6-aux ncurses
  2. Build it: cd /usr/ports/ports-mgmt/synth && make install
  3. Check with: synth configure
Synth should actually work without configuring, but check the default configuration to make sure it's going to use areas that have plenty of hard disk space.

It has not had widespread testing for FreeBSD yet. If you want to provide detailed feedback, the best place is GitHub ( https://github.com/jrmarino/synth/issues ) but if you don't have an
account you can just write back here and I'll check back periodically.

Known (current) limitations:
  1. It won't leverage pre-built official binaries at this time. It will build every dependency that it needs. I'll look into seeing how feasible downloading already-built dependency packages is in the near future.
  2. The "test mode" isn't complete. It doesn't check for leftover files or filesystem violations (the latter isn't a big deal because most of the build area is read-only so violators generally fail during the build).
  3. There's no "hang" watchdog yet. Any process that gets stuck won't be automatically killed and it could result in hanging mounts. For stock port trees, this shouldn't come up though, it's mostly a port developer issue.
I highly recommend that you review the README link above. There are screenshots! I hope people find Synth useful and a suitable replacement for PortMaster and PortUpgrade.
 
I'm very happy with my poudriere set up but I'm definitely going to play around with this.
 
As far as I understand there is nothing in it to guarantee clean build environments because no jail(8)s or chroot(8)s are used so pay special attention to /usr/ports/UPDATING for build time dependencies.
 
kpa said:
As far as I understand there is nothing in it to guarantee clean build environments because no jail(8)s or chroot(8)s are used so pay special attention to /usr/ports/UPDATING for build time dependencies.
Click to expand...

It uses chroot(8)s extensively. A clean environment is constructed for each and every port built. Execute "mount" command while synth in running and you'll see dozens of mounts. So yes, the environment is guaranteed to be clean and always the same on repeated runs.
 
marino said:
It uses chroot(8)s extensively. A clean environment is constructed for each and every port built. Execute "mount" command while synth in running and you'll see dozens of mounts. So yes, the environment is guaranteed to be clean and always the same on repeated runs.
Click to expand...

Ok :) The details of how it works just didn't jump at me at the first glance on the documentation. This sounds very promising, portmaster as it should have been from the get go.
 
I don't want to be too hard on Portmaster because it was written long before poudriere and pkg. Also, since I've never used portmaster myself (only portupgrade back in the day), when I reviewed the man page my eyes glazed over. I wondered how anybody could grok it quickly.

With that in mind, I intentionally made synth have a seemingly simple interface with a limited number of understandable commands. I aimed it at the everyday user, not the ports committer or expert. While it appears simple at the interface, it's got a lot going on underneath. I just didn't want to get too technical with the presentation, but rather preferred the goal of having it "just work" out of the box.
 
Out of curiosity, what is the gcc dependency for and why is it needed?
 
Crivens said:
Out of curiosity, what is the gcc dependency for and why is it needed?
Click to expand...
Synth is written in Ada.
gcc*-aux ports (gcc-aux, gcc5-aux, gcc6-aux) are versions of GCC that contain the Ada front-end. It's needed to compile the program (build dependency only)
 
Trying it out the first time I noticed that it auto-detected some settings from my /etc/make.conf such as DISTDIR. It did not auto-detect the ports tree directory PORTSDIR and I got an error about /usr/ports/ports-mgmt/pkg not existing on first synth configure run and I had to nullfs mount the ports tree at /usr/ports. Is it by any chance testing for the wrong variable?
 
If PORTSDIR is defined in the environment, it's picked up, with the highest priority. If it's defined in make.conf, it's not picked up. Basically it checks PORTSDIR (env), then /usr/dports, then /usr/ports and then it just defaults to /usr/ports. (this probably needs to eject before writing synth.ini and tell the user to set PORTSDIR).

You should use synth configure now to fix the portsdir setting and then you can remove the nullfs.

I'll put more graceful handling of this situation on the to-do list.
 
  • Thanks
Reactions: kpa
Yes, that's exactly what I did, it now uses my custom ports tree correctly after saving the configuration.
 
Okay, I just a private email with a question that revealed there's a bug for people with default setups.

On FreeBSD (but not DragonFly), the default location for distfiles is /usr/ports/distfiles. This directory passes the criteria as a ports category, so Synth will search it when scanning the ports tree, and quickly fails.

I need to update the Synth to avoid this directory.

If somebody wants an immediate workaround, it would be to move /usr/ports/distfiles to /my/new/location and set "DISTDIR=/my/new/location" in /etc/make.conf and run synth configure to change locations there as well. I'll push v98.1 soon though.
 
I've updated the port to version 0.98_2.
For systems that don't have $LOCALBASE/etc/pkg/repos directory, synth would fail to install anything, because the pkg conf file that synth installs (00_synth.conf) failed. The latest version creates the path if it's missing. (This can't happen on DragonFly, thus it took a FreeBSD user to flush out the oversight)
 
Couple of things. How can I use an existing /etc/make.conf to set the port options? It looks like synth is now creating its own /etc/make.conf under the chroot with preset options that can not be controlled. I really don't want to create the options under /var/db/ports because the saved options are a bad idea in my opinion.

Next is control-c and signal handling in general. It would be nice if the tool tried to clean up properly after being shut down by a signal, now it leaves the nullfs mounts behind if killed with control-c.
 
The man page and the graphic howto both address both those topics.

signal-handling: Use the "Escape" key. I had to abandon SIGINT capture because it propagates to the builders and breaks when conftests segfaults. You should almost never use control-C. However, if you do, run any major synth command, like synth configure and it will clean up the dirty mounts automatically.

See FILES on man page for make.conf customization (or miscellaneous on graphic howto).
 
  • Thanks
Reactions: kpa
Congratulations! Yesterday I tried it first time (yesterday it fails, but it seems it was something with my system. Today in the morning it runs fine). At the first look seems a fine tool.
What I am missing also on poudriere, I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).
 
talsamon said:
I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).
Click to expand...

That's what the synth status commands do. There are 3 flavors: synth status, synth status [ports], and (ignore this), synth status-everything.

See synth help, man 1 synth or the graphic howto on the GitHub site.
 
(to be clear, status just gives you the list that you want, it's a dry-run. You need to give the command to build to actually do anything)
 
talsamon said:
What I am missing also on poudriere, I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).
Click to expand...

I did suggest this to the poudriere developers but their response was that you can always use make all-depends-list (and the other two for run and build dependencies) either by running that in the jail or directly from the host.
 
kpa said:
I did suggest this to the poudriere developers but their response was that you can always use make all-depends-list (and the other two for run and build dependencies) either by running that in the jail or directly from the host.
Click to expand...

Except that tells you every dependency, not what is going to build.
If there are no packages, then "every dependency" is the same as the build list

If everything has already been rebuilt and only a couple of packages change, then maybe the build list is say, 3 packages long.
Synth status gives a list of what will build after evaluating which packages are still good and don't need rebuilding. So I don't necessarily agree with poudriere dev stance, if that's what was said.
 
Hi.

First, thanks for this and the work you put into it. It definitely seems like a large improvement on ports-mgmt/portmaster so far.

One question, is there a reason synth(1) doesn't nullfs(5)? mount the /usr/lib32 directory as well? At least on my machines running 11-CURRENT and 10.2-STABLE this is the case. Ports like emulators/virtualbox-ose will bail on building due to requiring access to the lib32 base libraries. I just left the configuration at the defaults for testing it out on both machines. I'm rebuilding all installed packages on both systems with it and so far haven't had a problem otherwise.
 
lib32 just wasn't implemented. The linux support is brand new but I forgot about lib32. DragonFly doesn't have these. I imagine it's not difficult to implement (e.g. one additional if on freebsdFreeBSD).

I ran into a potential issue with java/openjdk8 myself (socket timeout trying to connect to VM) so I'd be interested to hear if anyone has trouble with that one. I suspect the builders minimal /etc/services is missing a port/protocol there.
 
You must log in or register to reply here.
Back
Top