Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

To follow up: I upgraded my 10.1 amd64 system to 10.3-RELEASE.

Synth is working perfectly. No illegal instructions; everything looks nominal.
 
I've been using synth on 10.3 for a few days now without issues. The one complaint I have left is that I liked how portmaster, if you didn't use the -y flag, would stop before proceeding and show you what it would do. I know we can do this with synth status and don't know how many others would like that as default. (For example, _I_ like the fact that synth expects you to do all configuration before hand--that is, unlike portmaster, you don't get configuration dialogs by default, but others seem to prefer the portmaster way.)
 
scottro said:
I've been using synth on 10.3 for a few days now without issues. The one complaint I have left is that I liked how portmaster, if you didn't use the -y flag, would stop before proceeding and show you what it would do. I know we can do this with synth status and don't know how many others would like that as default. (For example, _I_ like the fact that synth expects you to do all configuration before hand--that is, unlike portmaster, you don't get configuration dialogs by default, but others seem to prefer the portmaster way.)
Click to expand...

It's not exactly ports-mgmt/portmaster's requirement to do configuration while running it, it's more of a consequence of it not setting BATCH by default. You can very well do all configuration beforehand and then just fire up portmaster(8) with (I hope I remember this right) -m -DBATCH and you won't see a single options dialog.
 
kpa thanks, I vaguely remember that now that you mention it, but at the time, whenever that time was, decided against it because there were a few ports that I wanted to configure.
 
This isn't exactly the right place to ask about package dependencies, but I'll put it the way it will concern synth functionality.

Now I have checked all of the installed ports on my system for the presence of any ports depending on lang/clang36 which used to be installed in the past as some build dependency and it pulled in devel/llvm36 with itself. I have deleted both, none of the currently installed ports depends on either of them, as per `make all-depends-list` command for each port (I wrote a script running this for each installed port).

Still, when I run synth prepare-system it starts building devel/llvm36 among other ones. I guess, it has long been replaced with devel/llmv37 which has already been built. So why does it insist on building llvm36? What does it know which I don't?
 
It comes from DEFAULT_VERSIONS that are in effect when building anything from scratch in a clean environment unless overridden by make.conf(5) or by some other means. Built packages are not known to the build process and can not influence what gets selected as a dependency, they are just pulled in if the already built package matches a build dependency selected by the ports infrastructure and DEFAULT_VERSIONS.
 
kpa said:
It comes from DEFAULT_VERSIONS that are in effect when building anything from scratch in a clean environment unless overridden by make.conf(5) or by some other means. Built packages are not known to the build process and can not influence what gets selected as a dependency, they are just pulled in if the already built package matches a build dependency selected by the ports infrastructure and DEFAULT_VERSIONS.
Click to expand...
I must have misunderstood your meaning somehow...
I've checked the file bsd.default-versions.mk, and neither clang nor llvm are even there.
And none of the already built|installed packages requires either lang/clang36 or devel/llvm36, this I specifically checked against all installed ports. Which also includes all build dependencies which I have kept.
 
Oh ok, I forgot something that affects the build just like DEFAULT_VERSIONS does. Many of the ports are using USES += compiler (/usr/ports/Mk/Uses/compiler.mk) in their Makefile and the selection of compiler to lang/clang36 comes from that.
 
kpa said:
Oh ok, I forgot something that affects the build just like DEFAULT_VERSIONS does. Many of the ports are using USES += compiler (/usr/ports/Mk/Uses/compiler.mk) in their Makefile and the selection of compiler to lang/clang36 comes from that.
Click to expand...
Is this not supposed to come out when you cd /usr/ports/your-port/dir && make build-depends-list ? And if not that, then make all-depends-list must get it at last, no?
 
Yes it should show up in the depends list but keep in mind that Synth's build environment may not be the same as the one you have on the host. Compare your /etc/make.conf (not read by Synth!) with /usr/local/etc/synth/LiveSystem-make.conf if you have one.
 
kpa said:
Yes it should show up in the depends list but keep in mind that Synth's build environment may not be the same as the one you have on the host. Compare your /etc/make.conf (not read by Synth!) with /usr/local/etc/synth/LiveSystem-make.conf if you have one.
Click to expand...
They're identical, cause I created them so.

It finally gave up building devel/llvm36 after N times it tried to and I had to interrupt it ungracefully. Hope I didn't break anything? There was some mentioning of it somewhere in the thread.

And know what? All this experience has encouraged me to look more carefully whether I actually need some other long-builders. Like www/webkit2-gtk2, which last time took unbelievable 5 hours to build! How funny it was to discover, then, that it was only needed because I unwisely checked "google" option for devel/gvfs . Which option I never used, nor was going to use any time in the future LOL, because gvfs is needed for fm/pcmanfm to mount SMB and other suchlike stuff.
 
...In fact, I actually needed neither of the webkits on my system. What a waste of time checking every option available!

Here is another aspect of how useful synth may be: watching the build process more closely you pay more attention to what's being built. Good to know for someone who cares about keeping things simple and uncomplicated.
 
It is not yet clear to me how to make synth install the built packages into the root defined in the profile as "system root directory". Because synth upgrade-system obviously upgrades synth's own system despite the "system root directory" setting, as I learned from my recent experience.
 
one thing doesn't have anything to do with the other. The system root is to define the build environment only.

You configure pkg(8) to use the new repository as necessary (e.g. another system connects to repository over a network or webserver). In that use case, pkg(8) is responsible for package installing. Synth only does it as a convenience command for the local system to save a step.
 
marino@ said:
one thing doesn't have anything to do with the other. The system root is to define the build environment only.

You configure pkg(8) to use the new repository as necessary (e.g. another system connects to repository over a network or webserver). In that use case, pkg(8) is responsible for package installing. Synth only does it as a convenience command for the local system to save a step.
Click to expand...
So you mean synth will imitate the things found under "system root" dir while creating its jailed build environment?

Yes, I was lacking clarity on this point :). I was thinking it was the "system root" as mentioned in synth upgrade-[I]system[/I].
 
There are also these words in the beginning of the man page:
Code: 
 synth will build packages in a clean environment can exactly mirror the system that
  they are built on...
This phrase is somewhat incomplete and needs correction. But apart from that, the word "can" sort of makes one think this is not the default behaviour "in a clean environment". But in reality this is what it will do if no list of packages to build is supplied (via commands that accept such option). For example, synth status (no pkg list given) will show the status of the build that will "exactly mirror" the current system. Then this behaviour can be called "default", right?

So maybe it would be clearer in the form like "...synth will exactly mirror the system that [the packages] are built on, unless ..." if that is what it will do anyway? Just to make it clear from the start, you know.

In public speaking it is a well known rule that the audience will remember most clearly what's mentioned at the start and in the end. That is why, while the introduction is always short, it must be precise in its terminology.
 
It's making a distinction between how poudriere does it and how synth does it.
Poudriere creates a reproducible clean environment. If you follow the same steps on 3 different machines (same arch) you'll get identical jails.

For some people, this is a problem because they have modified their host system and they want to build package for that exact system. By using a system root of "/" you get an exact mirror.

However, you can also dump another system (e.g. FreeBSD 9.3-RELEASE) in another location and set the profile's system root to that. Then the clean environment is identical to what poudriere will do.

so it is "can", not "will". The quoted text above is accurate. The default is "will", yes, because the default system root is "/", but that can be configured.

So you mean synth will imitate the things found under "system root" dir while creating its jailed build environment?
Click to expand...
Not imitate, mount. It mounts a selected set of directories like bin, sbin, usr/bin, usr/sbin etc. Others are created like /etc, /tmp, etc.
 
I want to use synth, instead of portmaster, in my automated script that bootstraps a fresh system.

So I installed the 10.3 release VM switched pkg to use latest and updated the ports tree with portsnap fetch extract. Ran pkg to bootstrap pkg.
At this point pkg is the only thing installed, ports tree is up to date but I do not know what is the correct way to bootstrap synth?

If I install synth with pkg and then run synth status, synth wants to rebuild synth and all of it's dependencies (gcc-aux,...).
If I install synth using ports and then run synth status it also wants to rebuild synth and all of it's dependencies (gcc-aux, ...).

So how do I install synth so that synth dose not want to rebuild it self?

Also how do I use synth in a script? I know I can have multiple profiles but how do I tell synth to use a profile without curses and without first running synth configure? Is there some switch option that can point synth to a profile file?

Thanks.
 
to bootstrap it, I suggest you download and install the officially built Synth package (using the bootstrapped pkg), then use Synth to rebuild Synth and ncurses and reinstall them.

See the SYNTHPROFILE environment variable how how to run a specific non-curses profile as a command.
The profile has to pre-exist though.
 
Clean install of 10.3. When preparing the system with Synth got the following error while building x11/qt5-x11extras:

Code: 
c++ -Wl,--as-needed -fstack-protector -fuse-ld=gold -Wl,--enable-new-dtags -pthread -Wl,-rpath,/usr/local/lib -shared -Wl,-Bsymbolic-functions -Wl,-soname,libQt5X11Extras.so.5 -o libQt5X11Extras.so.5.5.1 .obj/qx11info_x11.o  -L/construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/lib -L/usr/local/lib -lQt5Gui -lQt5Core -lGL
c++: error: invalid linker name in argument '-fuse-ld=gold'
*** [../../lib/libQt5X11Extras.so.5.5.1] Error code 1

make[3]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src/x11extras
1 error

make[3]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src/x11extras
*** [sub-x11extras-all] Error code 2

make[2]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src
1 error

make[2]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src
*** [sub-src-all] Error code 2

make[1]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1
1 error

make[1]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /xports/x11/qt5-x11extras

It looks that ld.gold is missing. Should I file PR?
 
You must log in or register to reply here.
Back
Top