Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

marino@ The system I have is a complete mess. I don't know why it happened, it just did, so I'm doing a re-install of everything from ground up (takes about 1 day now from start to finish). I don't know why I'm using v1.65. This is what make install installed when I ran that command. I suspect it might be related to the error mess I have been dealing with for the past few hours.
 
jonfr@ the version 1.65 is the latest, that's good.
As I said, if you are starting from scratch, remove all the options in /var/db/ports (assuming that is what you configured the options to be).
That will give you default options again.
If there are any ports you definitely want to customize, just use make -C /usr/ports/<category>/<port> config to set each one specifically.
 
I got one question about ports-mgmt/synth. It marks already installed packages (I install using make install clean so I can configure everything correctly) as new packages. I'm not ready to rebuild everything that has already been installed properly.

I get a output like this. Thanks for the help.

Code:
synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => print/indexinfo
  N => devel/gettext-runtime
  N => devel/gettext-tools
  N => devel/pkgconf
  N => devel/gmake
  N => databases/gdbm
  N => lang/perl5.24
  N => devel/libffi
  N => devel/p5-Locale-gettext
  N => devel/readline
  N => misc/help2man
  N => devel/xorg-macros
  N => lang/python27
  N => print/texinfo
  N => textproc/libxml2
  N => x11/xproto
  N => devel/m4
  N => devel/py-setuptools27
  N => textproc/expat2
  N => security/libgpg-error
  N => security/libgcrypt
  N => textproc/libxslt
  N => devel/libpthread-stubs
  N => devel/libcheck
  N => x11/libXau
  N => x11/libXdmcp
  N => x11/xcb-proto
  N => x11/libxcb
  N => x11/xtrans
  N => lang/python2
  N => x11-fonts/xf86bigfontproto
  N => devel/autoconf-wrapper
  N => x11/bigreqsproto
  N => x11/inputproto
  N => x11/kbproto
  N => x11/xcmiscproto
  N => x11/xextproto
  N => textproc/xmlcatmgr
  N => math/gmp
  N => devel/autoconf
  N => devel/automake-wrapper
  N => devel/libtool
  N => x11/libX11
  N => devel/automake
  N => graphics/png
  N => security/ca_root_nss
  N => devel/libevent2
  N => devel/py-pytz
  N => textproc/iso8879
  N => textproc/xmlcharent
  N => devel/bison
  N => devel/jansson
  N => devel/libev
  N => devel/py-babel
  N => print/freetype2
  N => textproc/docbook-sgml
  N => textproc/docbook-xml
  N => textproc/py-MarkupSafe
  N => textproc/py-pystemmer
  N => textproc/sdocbook-xml
  N => net/openldap24-client
  N => www/spdylay
  N => math/mpfr
  N => devel/py-Jinja2
  N => devel/py-six
  N => devel/scons
  N => security/krb5
  N => security/libssh2
  N => multimedia/librtmp
  N => dns/libidn2
  N => graphics/py-imagesize
  N => textproc/docbook
  N => textproc/py-alabaster
  N => textproc/py-docutils
  N => textproc/py-pygments
  N => textproc/py-snowballstemmer
  N => textproc/py-sphinx_rtd_theme
  N => archivers/liblz4
  N => archivers/lzo2
  N => www/nghttp2
  N => devel/binutils
  N => devel/cmake-modules
  N => devel/jsoncpp
  N => textproc/py-sphinx
  N => archivers/libarchive
  N => ftp/curl
  N => x11-fonts/fontconfig
  N => devel/cmake
  N => x11/libXext
  N => textproc/docbook-xsl
  N => x11/fixesproto
  N => x11/libXfixes
  N => misc/pciids
  N => devel/libedit
  N => devel/ninja
  N => x11/videoproto
  N => devel/libpciaccess
  N => devel/llvm37
  N => x11/damageproto
  N => x11/libXv
  N => devel/libclc
  N => devel/libdevq
  N => devel/makedepend
  N => x11/dri2proto
  N => x11/glproto
  N => x11/libXdamage
  N => x11/libXvMC
  N => x11/libxshmfence
  N => x11/presentproto
  N => graphics/libdrm
  N => devel/icu
  N => devel/pcre
  N => x11/libICE
  N => converters/libiconv
  N => graphics/libglapi
  N => devel/glib20
  N => x11/libSM
  N => x11/xf86vidmodeproto
  N => textproc/html2text
  N => x11/libXxf86vm
  N => x11/renderproto
  N => x11/xcb-util
  N => graphics/gbm
  N => textproc/minixmlto
  N => sysutils/gnome_subr
  N => x11-fonts/libfontenc
  N => devel/dbus
  N => x11/libXrender
  N => x11/pixman
  N => x11/xcb-util-renderutil
  N => graphics/libEGL
  N => graphics/libGL
  N => x11-fonts/mkfontscale
  N => devel/nasm
  N => security/libtasn1
  N => security/nettle
  N => emulators/tpm-emulator
  N => dns/libidn
  N => graphics/cairo
  N => textproc/p5-XML-Parser
  N => x11-fonts/mkfontdir
  N => devel/dbus-glib
  N => devel/gobject-introspection
  N => devel/libdaemon
  N => security/p11-kit
  N => security/trousers
  N => graphics/jpeg-turbo
  N => textproc/intltool
  N => security/gnutls
  N => graphics/jbigkit
  N => net/avahi-app
  N => print/libpaper
  N => graphics/tiff
  N => print/cups
  N => graphics/giflib
  N => archivers/unzip
  N => x11/libXi
  N => x11/recordproto
  N => java/java-zoneinfo
  N => archivers/zip
  N => x11-fonts/dejavu
  N => x11/libXtst
  N => x11-toolkits/libXt
  N => java/bootstrap-openjdk
  N => java/javavmwrapper
  N => audio/alsa-lib
  N => print/gsfonts
  N => java/openjdk7
  N => graphics/jbig2dec
  N => graphics/lcms2
  N => graphics/svgalib
  N => graphics/webp
  N => shells/bash
  N => math/mpc
  N => devel/talloc
  N => print/ghostscript9-agpl-base
  N => java/openjdk8
  N => graphics/gd
  N => graphics/jasper
  N => textproc/py-libxml2
  N => lang/gcc6-aux
  N => devel/apache-ant
  N => devel/ncurses
  N => devel/p5-Parse-Yapp
  N => devel/popt
  N => devel/tevent
  N => graphics/netpbm
  N => graphics/peps
  N => graphics/scr2png
  N => textproc/docbook-xsl-ns
  N => textproc/igor
  N => textproc/iso-schematron-xslt
  N => textproc/itstool
  N => textproc/scr2txt
  N => textproc/xhtml
  N => databases/tdb
  N => archivers/p7zip
  N => www/links1
  N => x11-fonts/droid-fonts-ttf
  N => x11-fonts/gentium-plus
  N => x11-fonts/lohit
  N => misc/freebsd-release-manifests
  N => misc/ini_file_manager
  N => devel/adacurses
  N => devel/gamin
  N => devel/libinotify
  N => devel/p5-Parse-Pidl
  N => dns/py-dnspython
  N => textproc/docproj
  N => textproc/fop
  N => japanese/font-ipa
  N => chinese/arphicttf
  N => databases/ldb
  N => databases/ntdb
  N => korean/nanumfonts-ttf
  N => sysutils/libsunacl
  N => misc/freebsd-doc-en
  N => devel/py-iso8601
  N => ports-mgmt/dialog4ports
  N => ports-mgmt/poudriere
  N => ports-mgmt/synth
  N => dns/dnsmasq
  N => net/dhcpd
  N => net/samba42
  N => net/whois
  N => editors/nano
Total packages that would be built: 226
The complete build list can also be found at:
[/U]
 
I'm not ready to rebuild everything that has already been installed properly.

It seems you've misunderstood what Synth is and what it does. The purpose of Synth is to create a local, custom package repository. While it is capable of installing individual ports or upgrading installed ports, it does so by first updating the local repository, then installing or upgrading packages using that repository. It is not simply a front-end to the ports system; before it can do anything at all, it must create the package repository. You therefore need to build everything the first time around.

The only possible way around that is to set "Fetch prebuilt packages" to "true," which will fetch packages from the official repository when possible. But even that will only fetch packages which you have not customized, and which are not affected by any customizations you have made---meaning that it will almost certainly have to rebuild some ports that are already installed, and if you've set any global customizations using make.conf(5) you'll probably have to build almost everything anyway.
 
It's called "Synth" and one of its main purposes is to replace portmaster
Sorry for not able to read all 33 pages, however I have a question.
I'm thinking to replace portmaster on my system, however when I tried synth I found it's lacks the ability to ask for port options. So, basically, it's worse than portmaster for me as I need to run make config first. I don't want to do it for all run-deps manually. Are you planning to add interactive mode like portmaster has?
Thanks.
 
no, I'm never adding an "interactive" mode. It's not needed (despite what you may believe). In reality, people only change the options on a handful of ports. For all the rest they use the defaults. Synth uses the stored defaults (if any) or the defaults. It works fine.

You're just stuck thinking that the "portmaster way" is the "correct way". You have to expand your thinking.
In your case it would be easier to delete all the currently stored option (because you're sure to have dozens of obsolete ones, courtesy of portmaster) and redo any that need to be changed from defaults. Global changes you can do through the profile -make.conf file.
 
Stored options are pain in my opinion, I prefer to set everything in the builder's make.conf like this:

Code:
OPTIONS_UNSET= CUPS DBUS X11

DEFAULT_VERSIONS+= ssl=openssl

ports-mgmt_poudriere-devel_SET= ZSH

#devel/git
devel_git_UNSET = SEND_EMAIL

# editors/vim
editors_vim_SET= CONSOLE
editors_vim_UNSET= RUBY TCL ATHENA GNOME GTK2 MOTIF

And so on. This is easier for me because of all of the options are now centralized into one single file that I can edit and update in one go. The downside is that there's a bit of extra initial work to get a list of the options in a format suitable for the make.conf file.
 
In reality, people only change the options on a handful of ports.
I change a lot of options. I know what they do and what I need on my laptop. If I don't I'd probably switched to packages. Can you explain what is "portmaster way"? I can delete all port options and set them again, that's not the problem, but I don't understand how can I do this again initially (and maintain) with synth? Looks like reinvention of poudriere in more solid and simple way. And no, I am not trolling, but I see that portmaster is not maintained for years and I need to know what to do when it finally stops completely.

Even If one need to change 1 port option, he *must* run make config. What if this is run dep? We must start building with investigating of dep tree and how changes affecting it? I'd say synth is impossible to use in portmaster way, so it's impossible to replace it.

I prefer to set everything in the builder's make.conf like this
Gentoo way If I recall their system correctly. However the problem of knowing exact dep tree exists for this option as well. For example, if you want to install something new, you are forced to do it not in synth, because you need to gather initial options somehow. And to maintain - if option list changes.
 
step 1) rm -rf /var/db/ports (or whatever directory you use for storing ports options)
step 2) identify the ports that you have different options
step 3) make -C <category>/<port> config for those identified in step 2
step 4) build the first set of packages given a specific build list or querying the host system with "synth prepare-system"
step 5) periodically repeat step 4. Generally you don't touch options again.

that's it.
You don't have to worry about anything else, you don't have to worry about recursion or anything.

It's a "replacement" of functionality. It's quite possible to replace portmaster, many already have.
 
note that you don't have to delete old options. You can set PORT_DBDIR in /etc/make.conf to a new directory and then make synth use that new directory after you set the options.
 
Stored options are pain in my opinion, I prefer to set everything in the builder's make.conf like this.

The downside of this approach is that Synth can't detect when the options are obsolete. When the option set is saved as a snapshot it's easy to determine this, which alerts the user that the port options have changed.
 
step 2) is a tricky one as I don't know what was changed. What would you suggest? Personally, I'm thinking to config-recursive all top level ports and set 'pristine good options' directory for synth to use.
 
well, it's been recommended NOT to do that. The recommendation is only set options on packages that need options that differ from default.

You can use /usr/ports/Tools/scripts/redundant-opt-files.sh to remove all the redundant options files you currently have. what's left are either "bad" saved options (which synth will detect) and the changed ones which you want.

In the end, you are free to use the tools in non-recommended ways though. It's at your own risk.
 
  1. Insist on building ports from source no matter what.
  2. Need ports with options other than defaults.
  3. Want system upgrades to be painless.
  4. Have multiple systems that they want to share a common, customized
    package repository.
  5. Think postmaster is great.
  6. Want to gain concurrent package building locally.
  7. Don't like package surprises.
  8. Don't understand or want to deal with poudriere.

Thank you so much for your work on this - it's awesome!

I answer YES to all of the above except #5 - portmaster has not been pleasant in recent months with FreeBSD 11-Release and I am not happy with failed port updates using /usr/local/sbin/portmaster -a -d --no-confirm -G in my custom script that updates everything. Even adhering to /usr/ports/UPDATING doesn't help prevent numerous 'gotcha's' that just happen when updating ports. I'm tired of spending time manipulating port installs that want python 2 or python 3, rust 12 or rust 13, can't figure out which libboost to use and so on. #1 above is virtually impossible with portmaster now.

I also think it's great to see something written in Ada! I took Ada in college ages ago (Janus Ada) and thought it was before it's time. Ada is an excellent choice for Synth. Back then, we avoided Ada at all costs and went with C :)

Finally, portmaster holdouts - if you are wondering about synth, the switch over is not painful at all.
 
read the man page or instructions on github regarding [profile]-make.conf.
/etc/make.conf is NOT used, every profile has their own profile-specific file that it uses instead.
 
read the man page or instructions on github regarding [profile]-make.conf.
/etc/make.conf is NOT used, every profile has their own profile-specific file that it uses instead.
I know about this.
Code:
cat /usr/local/etc/synth/LiveSystem-make.conf
DEFAULT_VERSIONS+=ssl=libressl perl5=5.24 pgsql=9.6
OPTIONS_UNSET= DOCS EXAMPLES IPV6

.if ${.CURDIR:N*/ports/security/strongswan} == ""
DEFAULT_VERSIONS=ssl=base
.endif
For general make.conf this code should work, however if is ignored when port being build under synth.
 
that's not legal. You can't have different ssl versions. It's either base or something else, but a mixture is wrong (by definition). It's not a matter of opinion, this has been stated quite a few times.
 
that's not legal. You can't have different ssl versions. It's either base or something else, but a mixture is wrong (by definition). It's not a matter of opinion, this has been stated quite a few times.
strongswan with libressl has issue and not working at all.
I already reported it https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212149 and use linking against base as temporary solution. I suppose I should make a different profile for that port?
 
I am saying I am not sure your temp solution even functions. There's a reason default ssl is set once, the different versions can conflict (built against one and pulls in the other, libraries affected etc.).

Your hack doesn't work because your pattern is wrong. It would be "N/xports/security/strongswan". The ports aren't put in /usr/ports internally.
 
I am saying
I'm using it for 2 months. Yes, I understand that it's not very good solution and hope switch to openiked which was recently added to port tree.
Thanks for the hint, I should notice it in MAKE_ENV section of synth logs.

BTW, synth acts better here than portmaster - I patched Mk files to break openssl conflict detection, it's not necessary for synth.
 
Well, I finally learned of this software and gave it a try. I'll share my findings and leave it at that.

I can say: it's not for me. First of all I dislike some of the approach which I pick up as plain out arrogant. It even shows in the OP: "Aimed at people who think Portmaster is great" as well as in the synth(1) manual page: "It was hoped that it would attract users of the older PortMaster and PortUpgrade ports management tools by providing them with a superior and well-maintained alternative.". Call me old fashioned but approaches like these tick me off. Live and let live is my motto, which means showing respect for other people's work.

Unfortunately for me this also sets a mindset.

So here I am in /usr/ports/devel/apache-ant and I want to check which ports Synth found to be new. Picture me surprised...

Code:
peter@macron:/usr/ports/devel/apache-ant# synth status
Please change the current directory; Synth is unable to launch from here.
As more experimentation showed me: Synth refuses to run inside the /usr/ports structure. I'm sure there are good reasons for that, but it's not always very desirable nor user friendly. Especially when using commands which do very little but show me the current status.

But more so because I sometimes use Portmaster for specific tasks. Including, if needed, a careful upgrade of one specific port (-o). Which means that I'll have to specify the origin. Doable by utilizing #pkg info -ox <name> but also by being in the physical directory itself. That of course doesn't work.

Of course I agree that this is but a detail, and I'm also not ignoring the possibility that there are most likely good reasons for this. Obviously it is important to use a tool in a proper and sometimes intended way. Still, I think it's fair to mention nonetheless given the direct comparison with Portmaster regarding "superiority" in the overall (which, for me, includes user friendliness).

So then I tried this again, this time in the proper directory called /root/updates where I keep some of my custom made scripts (utilizing Portmaster). This time I was greeted with another error which made me seriously scratch my head:

Code:
peter@macron:~/updates#synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs tmpfs /usr/obj/synth-live/SL09 => failed with code 1
Why in the world would Synth rely on a (memory wasting) temporary file system while /tmp and /var/tmp have been carefully set up for this very specific reason? This machine I'm using isn't the most modern one, but it can maintain a package repository just fine.

Sure, I may have completely ignored any required steps to set up and configure Synth. But there's no mentioning of this requirement in the manual page. Obviously I learned about /usr/local/etc/synth/synth.ini, also because I was curious about the tmpfs approach and this file does have some possibly related options (Tmpfs_workdir and Tmpfs_localbase). Of course all the synth(1) manualpage tells me is that I don't have to touch this file, and the file itself doesn't have a manualpage at all.

So I learned of # synth configure, used that and confirmed my suspicions: The options K and L which provide me with a way to make Synth use (or not use) tmpfs for the work area and localbase (or both of course).

Just too bad that these options apparently get totally ignored. It's the only conclusion I can come up with.

So yeah, each to their own but this is definitely not for me. Synth might be more extensive feature wise (I simply cannot say, I only base myself on what I read in the manualpage) but for me Portmaster can take care of all the tasks I need while also ensuring that I remain fully in control over what it can and cannot do. For example the option to ensure that I always have a backup package the moment it upgrades a port. I don't see anything like that mentioned in the Synth documentation. It might be a supported feature, and I cannot rule out the option that I overlooked it, but for me that also adds up to user friendlyness.

I personally prefer the Unix mindset where a program performs a small task which allows for it to become part of a bigger task. Which is exactly what Portmaster provides. It probably takes more work to configure and set it up, no arguments there, but Portmaster easily allows me to set up my own package repository and distribute those to my other servers.

It might not be an "all in one" solution, but that definitely does not make it inferior, as hinted at in your documentation.

Alas, I do wish you good luck with the project and hopefully there's still something useful in here besides my (positively meant) criticism :)
 
I can say: it's not for me. First of all I dislike some of the approach which I pick up as plain out arrogant. It even shows in the OP: "Aimed at people who think Portmaster is great" as well as in the synth(1) manual page: "It was hoped that it would attract users of the older PortMaster and PortUpgrade ports management tools by providing them with a superior and well-maintained alternative.". Call me old fashioned but approaches like these tick me off. Live and let live is my motto, which means showing respect for other people's work.

What you define "arrogant", is what I define "truth", and until now I have followed all threads and some PR too regarding the comparison, and what marino@ say is simply the truth, and to my knowledge no one until now has posted some fact to demonstrate that marino@ is wrong.

Instead this forum is plenty of threads which demonstrated that very often portmaster / portupgrade doesn't do the right job., it was also already mentioned that portmaster was abandoned from the original author and just forcefully maintained now by someone else.

About option K / L and tmpfs, while Synth allow to use or not tmpfs for these two areas, there are of course other places where tmpfs are used without questions, so it is not that the options are ignored, it is that tmpfs are used elsewhere for other things, ultimately for performance reasons.

I hope you will give Synth another try, while Synth has a simple interface, it is not a simple tool and do lot of things better than portmaster, starting from building in a clean environment up to building a whole repository (or more repositories if needed).

By doing all that Synth effectively may be somewhat heavier from a system usage point of view, but also it perform lot of checks on behalf of the users to assure the quality of the results. (detecting circular dependency is just one case that come up to my mind, there are other of course).
 
Synth is awesome. It can abort on some old slow cpu machines when building certain ports, and for that reason I think portmaster needs to stay the official tool, but other than that Synth is the slickest thing since round wheels.
 
Back
Top