Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

Anything like @FOO_BAR@ is supposed to be filled in by autotools (devel/autoconf etc.) so it's not detecting our /usr/bin/strip for some reason. There is STRIP_CMD provided by the ports system but it's supposed to be used in post-install. You'll probably have to hack the configure script (and that genmod.sh script) anyway to either fix the @TARGET_STRIP@ part or replace it with our own STRIP_CMD that is done in post-install.

https://www.freebsd.org/doc/en/books/porters-handbook/install.html
 
OK, don't bother. It's been fixed upstream.

I discussed this for some time with GRUB2 developer team and it was because our CURRENT uses a rather old version of strip & friends. But then they upgraded their binutils to the latest version and it behaves the same way as our strip. So they finally had to fix this and I don't need to bother, just update my port to the latest GIT.
 
kpa said:
I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:

Code: 
BUILD_DEPENDS+= ${LOCALBASE}/bin/python:lang/python
Click to expand...

using PORTSDIR optionally isn't done operationally yet (i think it's because the quarterly branch doesn't know it and that would cause problems if the trunk ports removes it and then the port is backported).

I think the tree is going to be actively switched to no PORTSDIR when the next quarterly is branched, but until then even new ports will/should still have it.
 
  • Thanks
Reactions: kpa
Since sysutils/py-salt is broken because lang/python27 is built against
an old version of security/openssl, as this error seems to suggest

Code: 
pedja@freebsd102:/usr/ports/lang # salt
Traceback (most recent call last):
  <snip>
  import ssl
  File "/usr/local/lib/python2.7/ssl.py", line 97, in <module>
  import _ssl  # if we can't import it, let the error propagate
ImportError: /usr/local/lib/python2.7/lib-dynload/_ssl.so: Undefined symbol "SSLv2_method"
I wanted to rebuild all the dependencies for Salt.

After some googling and reading synth(1), I came up with this:
pkg query %do py27-salt-2015.8.7 | awk -vORS=" " '1' | xargs synth force

But, Synth exits with the error just after the build of the packages.
Build log:
Code: 
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
py27-pytz-2015.7,1.txz failed dependency check.
py27-six-1.9.0.txz failed dependency check.
py27-Babel-2.2.0_1.txz failed dependency check.
py27-backports_abc-0.4.txz failed dependency check.
py27-dateutil-2.5.0.txz failed dependency check.
py27-jmespath-0.9.0.txz failed dependency check.
py27-singledispatch-3.4.0.3.txz failed dependency check.
py27-docutils-0.12.txz failed dependency check.
py27-certifi-2015.11.20.txz failed dependency check.
Scanning existing packages.

The task is complete.  Final tally:
Initial queue size: 25
  packages built: 25
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:05:52
The build logs can be found at: /var/log/synth
Would you like to rebuild the local repository (Y/N)?

raised ADA.IO_EXCEPTIONS.END_ERROR : a-textio.adb:608

Should I just nuke the synth repository and start from scratch?
Or I am just DoingItwrong[tm]?

Synth-1.11, FreeBSD-10.3-BETA3, latest ports revision.

Thank you.
 
try run the same command but with "just-build" instead.

"raised ADA.IO_EXCEPTIONS.END_ERROR : a-textio.adb:608" is showing because it's asking a question and there's no terminal (apparently). Since you already deleted the packages, "just-build" should work. it doesn't ask questions. Probably synth needs to detect a missing terminal and skip the questions.

OR

just run synth rebuild-repository or synth upgrade-system
It looks like it finished building. you don't need to feed it any more ports.
 
marcpearsonti said:
If Portmaster can show the versions when showing the "status", Synth should be able to too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg(8) version could be great ! But I'm not a pro !
Click to expand...

Hi Marc,
I added this functionality ( https://github.com/jrmarino/synth/commit/276d8309dab8aa9c95c3b165fb3de614e8df980b ) and it will be available on the next release of Synth.

Note that the version display compares what is packaged in the profile's package directory NOT what is installed on the host system. If you think about it, this makes sense. (usually what's packaged is also installed but there's no rule where that is always true).

There are two status lists, one for the screen and one logged in the /tmp directory. The screen version shows N/R/U (for New package, Rebuild existing package (same version) and Upgrade package (new version)). For the latter, the old version and new version are also shown on the screen. For the log, versions are also shown for "New" and "Rebuild" entries. I didn't show them on the screen because it made the display hard to read. Anyway, I think you should be happy with the implementation.
 
(Apologies for the late reply, I was debugging libvirt, so the FreeBSD VM was offline for a while).

Anyway, synth upgrade-system didn't work, for some reason
(synth got so confused at one point that on consecutive runs either built some packages, or reported that all is OK.)
So, in the spirit of the old adage 'When in doubt, nuke.', I did just that.

Default configuration, synth prepare-system did its thing, all good.
And I think I know why synth upgrade-system doesn't reinstall the freshly built packages.

If I am understanding pkg.conf(5) correctly, because of the
Code: 
CONSERVATIVE_UPGRADE
pkg() prefers to use the repository from which the packages were originally installed (upstream FreeBSD, in my case), and since
I am trying to reinstall and not update, it just does what its supposed to do when this option is set.

What I do not understand is how and when is the repository priority evaluated?
Just on the package update (when the package in Synth repo has higher port version/revision number than upstream),
or if the repo has higher priority than the official one, then it is used to update and reinstall packages, provided it has the package and
all its dependencies)?
(Also, how high does the priority number go?pkg.conf(5) just says "Higher values are preffered, default is 0.")

So, Synth helpfully built all the packages I needed, now to figure out how to actually install them, preferably without
breaking something :)

Off-topic: marino@, I very much enjoyed your interview on BSDNow, particularly the story about Ada.
Question, if I may: what does
Code: 
Building pkg
when synth prepare-system is run actually do?
I haven't had the chance to look at the source yet.

I'll stop here with the questions :)
(I am new to FreeBSD, so naturally I have a lots of them).
 
Pedja said:
If I am understanding pkg.conf(5) correctly, because of the CONSERVATIVE_UPGRADE
pkg() prefers to use the repository from which the packages were originally installed (upstream FreeBSD, in my case), and since
I am trying to reinstall and not update, it just does what its supposed to do when this option is set.
Click to expand...

Normally conservative upgrade is only an issue if you run pkg upgrade manually. internally synth tells package to use the synth repository exclusively for upgrades. But I'm not sure I agree with option's default, you should probably disable it.

What I do not understand is how and when is the repository priority evaluated?
Just on the package update (when the package in Synth repo has higher port version/revision number than upstream),
or if the repo has higher priority than the official one, then it is used to update and reinstall packages, provided it has the package and
all its dependencies)?
(Also, how high does the priority number go?pkg.conf(5) just says "Higher values are prefered, default is 0.")
Click to expand...

As far as I can tell, "priority" doesn't work. I think it's a decoy (aka bug). The only way I can get a repository to be taken preferentially is to define the conf file lower alphabetically. this is why the conf file is named 00_synth.conf, which should come before any others.

So, Synth helpfully built all the packages I needed, now to figure out how to actually install them, preferably without
breaking something :)
Click to expand...

method 1: synth upgrade-system
method 2: (assuming 00_synth.conf already created) pkg upgrade -r Synth (or something similar -- -r limits to synth repo)

Off-topic: marino@, I very much enjoyed your interview on BSDNow, particularly the story about Ada.
Question, if I may: what does
Code: 
Building pkg
when synth prepare-system is run actually do?
I haven't had the chance to look at the source yet.
Click to expand...

Thanks. synth prepare-system gets pkg(8) to return a list of all installed ports, and pipes that list into synth (think of it as a list of port origins), it incrementally builds what is necessary, and when it's done, it rebuilds the repository.

The idea is that somebody would run pkg upgrade -r Synth manually so they can be present during the sytem upgrade (or it's building a repo for clients other than the localhost). Basically it constructs the repo without changing the localhost.
 
Last edited by a moderator:
Oh, you mean why is it building pkg(8)? pkg(8) is needed inside the builders for all sorts of reasons. It's the first package built, always. Poudriere does the same thing.
 
Last edited by a moderator:
marino@ said:
<snip>
method 1: synth upgrade-system
method 2: (assuming 00_synth.conf already created) pkg upgrade -r Synth (or something similar -- -r limits to synth repo)
<snip>
Click to expand...
pkg upgrade -fr Synth worked.
I was a bit concerned because it downgraded a few packages, but
when I ran pkg update after, those packages were upgraded, too.

So, all is OK now (famous last words...)

Thanks for such a prompt reply, btw :)
 
marino@ said:
... If you use ccache, it will rebuild huge ports in minutes.
Click to expand...
Doesn't seem to effect www/webkit2-gtk3 this way. It's obviously just rebuilding the port due to some modifications in some other ports, yet the build is taking the same huge amount of time it did originally (already 1hr 30 minutes have passed, the same slow pace).

Are there any other ccache settings to take into account that would really speed up the build process? My ccache size limit is set to be 20G.
 
free-and-bsd@
I saw webkit2-gtk3 take 15 minutes to build yesterday, with ccache.

The most likely explaination is that you haven't configured synth for ccache correctly. How *exactly* (be detailed) did you do it?
 
marino@ said:
free-and-bsd@
I saw webkit2-gtk3 take 15 minutes to build yesterday, with ccache.

The most likely explaination is that you haven't configured synth for ccache correctly. How *exactly* (be detailed) did you do it?
Click to expand...
Thanks for prompt reply!
I did as you suggested earlier: in synth configure I inserted the path to ccache dir where synth wanted it. It's running now, so I can't show configuration.
As for ccache itself, sudo ccache -p shows:
Code: 
(default) base_dir =
(default) cache_dir = /root/.ccache
(default) cache_dir_levels = 2
(default) compiler =
(default) compiler_check = mtime
(default) compression = false
(default) compression_level = 6
(default) cpp_extension =
(default) direct_mode = true
(default) disable = false
(default) extra_files_to_hash =
(default) hard_link = false
(default) hash_dir = false
(default) log_file =
(default) max_files = 0
(/root/.ccache/ccache.conf) max_size = 20.0G
(default) path =
(default) prefix_command =
(default) read_only = false
(default) read_only_direct = false
(default) recache = false
(default) run_second_cpp = true
(default) sloppiness =
(default) stats = true
(default) temporary_dir =
(default) umask =
(default) unify = false
So I typed in /root/.ccache as ccache dir for synth.
 
With root on another tty, type ccache -s
You should see the statistics change as synth builds (run command repeatedly). If not, it's not working.

Remember you get the time savings on the second run. the first build is normal time.
 
Firstly, thanks for synth. One thing I haven't seen. (I haven't read every post in the thread, but I've looked at several, and tried to find this in the man page with no luck).

With portmaster, if I wanted to add an option at build time--such as with dwm, the window manager, I would run

portmaster -m 'DWM_CONF=/usr/home/scott/dwm/conf.h install x11-wm/dwm.

Is there something similar for the -m to add options in synth that I've overlooked?
 
marino@ said:
with root on another tty, type ccache -s
You should see the statistics change as synth builds (run command repeatedly). If not, it's not working.

Remember you get the time savings on the second run. the first build is normal time.
Click to expand...
Yep, it is changing. Well maybe you're right and I set ccache after the first build of webkit2-gtk3 took place. I remember starting and stopping synth multiple times before the original repo got finished.
I'll check this once I'm finished with this rebuild.
 
scottro said:
Is there something similar for the -m to add options in synth that I've overlooked?
Click to expand...
looking up in the portmaster man page:
Code: 
-m    arguments for make

The answer is no, synth will not let you do this (frankly that looks dangerous to me).
Your only option for customizaton beyond configuring port options is to use a [profile]-make.conf

obviously you can set
Code: 
DWM_CONF=/usr/home/scott/dwm/conf.h
in /usr/local/etc/synth/LiveSystem-make.conf and x11-wm/dwm will use it, but so will every other port.
 
free-and-bsd said:
Yep, it i s changing. Well maybe you're right and I set ccache after the first build of webkit2-gtk3 took place. I remember starting and stopping synth multiple times before the original repo got finished..
Click to expand...

webkit2-gtk3 is quirky. It seems to forgot the cached bits --- almost like something invalidates the whole thing. I know I had built it under ccache, and yesterday I built it 3 days. The first time was 2.5 hours, as if it had not been cached. I am pretty sure it was, but it didn't work. The next run it did though. Maybe something in the source code (header change) invalidated the whole thing.
 
Thanks for the quick answer. It's not a problem, I can simply do that one package with the standard cd into the port directory and running the make option there.
 
You must log in or register to reply here.
Back
Top