Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

[talsamon]

Could someone please testbuild deskutils/calibre

Crivens confirm, hangs on

Installing bash completion to: /ram/usr/ports/deskutils/calibre/work/stage/usr/local/share/bash-completion/completions/calibre
Setting up desktop integration...

but hangs also in the port on my system - maybe, it is something with ccache?

 

[xtaz]

Just tried that. It doesn't log anything at all to suggest it is even trying to?

Stand by, building pkg(8) first ... done!
Stand by, updating external repository catalogs ... done.
pkg: Repository Synth missing. 'pkg update' requiredpkg: Repository Synth missing. 'pkg update' required

pkg: Repository Synth missing. 'pkg update' required
pkg: Repository Synth missing. 'pkg update' required
etc etc etc....
Scanning existing packages.
00:00:04 => [03] Builder launched
00:00:04 => [02] Builder launched
00:00:04 => [01] Builder launched
00:00:14 => [02] 00:00:06 Success devel/autoconf-wrapper
00:00:14 => [03] 00:00:06 Success devel/automake-wrapper
00:00:25 => [01] 00:00:17 Success devel/ccache
00:00:26 => [03] 00:00:09 Success misc/mime-support
00:00:35 => [03] 00:00:07 Success print/indexinfo
00:00:56 => [03] 00:00:17 Success textproc/expat2
00:01:04 => [02] 00:00:47 Success devel/cmake-modules

And so on..... Would you expect to see something else there then?

 

[xtaz]

Ahhhh I see the problem. If I delete /usr/local/etc/pkg/repos/00_synth.conf then it fetches them all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn't been built yet. Chicken and Egg situation?

Both repos appear to have priority 0.

 

[marcpearsonti]

Hello marino@

Excuse my English, I'm a French speaker.

First, I would like to thank you for Synth, is a great tool

But I have so many questions ...

1. How Synth scan the port tree ? I get the port with SVN from 2016Q1 branch. But it's missing ports-mgmt/synth in it. So I put Synth manually in the port tree with a specific checkout from the head but I got the error when I execute synth status:

Scan of ports-mgmt/synth failed, it will not be considered.

2. Why Synth need to build so many ports when a package like ftp/curl change. I have used portmaster(8) since many years and it doesn't do this kind of rebuild. My FreeBSD server never broke. Maybe the way Synth works is the good one but I would like to understand. Maybe I was lucky with portmaster(8)

3. synth status shows that 3 ports need to be rebuilt but after the build/install process is complete, synth status show the same 3 ports with failed dependency check. How can I debug this situation ?

4. When Synth rebuilds the package repository, does it need to repackage all of them or only the ones who changed? On my old FreeBSD server this operation take approx. 30 minutes for 312 packages

5. Is the message "<portname> failed dependency check" how Synth report that a package needs to be rebuilt or is it an error message?


I would like to make suggestions for improvements:

1. It's possible when running synth status to show from old version to new version each ports will be upgrade. Example: databases/rrdtool (4.8.9_1 to 5.0.1)
2. If "failed dependency check" is a normal message, is't a little bit scary. You should use a message like "<portname> is no more in sync with dependencies" (You are better in english then me!)
3. Could Synth keep old versions of a package (Configure could have a yes/no option and how many old versions to keep ). So if a user need to rollback a package it will be very easy.

 

[marino]

hem all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn'

Ahhhh I see the problem. If I delete /usr/local/etc/pkg/repos/00_synth.conf then it fetches them all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn't been built yet. Chicken and Egg situation?

Both repos appear to have priority 0.

Don't know. IIRC it was told to only look at the external repo (which appears to update successfully).
It's not looking for itself, but that strange error might be preventing it from working.

 

[xtaz]

The strange errors are because the 00_synth.conf file is being read but because it is synths very first build run none of the metadata files exist inside the repo directory so it's not (yet) a valid repo to use. I guess in that situation it isn't falling back to using the FreeBSD repo?

 

[marino]

But I have so many questions ...

It's probably better if you asking them one at a time, this is a bit much.

1. How synth scan the port tree ? I get the port with SVN from 2016Q1 branch. But it's missing ports-mgmt/synth in it. So i put synth manually in the port tree with a specific checkout from the head but i got the error when i execute synth status : Scan of ports-mgmt/synth failed, it will not be considered.

You've taken a known good tree and added something to it, and now you got an error. You should probably expect that when you transplant ports from one tree to another . (Don't do it). I could guess it failed because gcc6 isn't in the Quarterly tree or something. Don't corrupt trees anymore is my advice.

2. Why synth need to build so many ports when a package like ftp/curl change. I have used portmaster since many years and it doesn't do this kind of rebuild. My FreeBSD server never broke. Maybe the way synth works is the good one but il would like to understand. Maybe i was lucky with portmaster

Yes, you were lucky. (And I kind of doubt you never had a problem). hundreds of packages depend on Curl so you should expect major fallout. This very port has been discussed earlier in the thread.

3. synth status shows that 3 ports need to be rebuild but after the build/install process is complete, synth status shows the same 3 ports with failed dependency check. How can I debug this situation ?

Define WHYFAIL in the environment with synth(1) in non-curses mode. It will tell you exactly why. It could be a bug in the port.

4. When synth rebuild the package repository, does it need to repackage all of them or only the ones who changed? On my old FreeBSD server this operation take approx. 30 minutes for 312 packages

This question doesn't make sense. "pkg repo" packages *ALL* of them, by definition. You can't do it any other way.

5. Does the message "<portname> failed dependency check" is how synth report that a package need to be rebuild or is a error message ?

it will delete the package (unless it's a status command).
This isn't an "error" message, it's normal.

I would like to make suggestions for improvements
1. It's possible when running synth status to show from old version to new version each ports will be upgrade. Example: databases/rrdtool (4.8.9_1 to 5.0.1)

No, it's not possible (nor do I see the value)

2. If "failed dependency check" is a normal message, is't a little bit scary. You should use a message like "<portname> is no more in sync with dependencies" (You are better in English then me !)

Why is that scary? It's normal.

3. Could synth keep old versions of a package (Configure could have a yes/no option and how many old versions to keep ). So if a user need to rollback a package it will be very easy.

This also makes no technical sense, it's not possible.
You're creating a local repository, everything is integral, you can't just roll some random port back.
Let's put this another way: Have you seen any tool that does that?

 

[marino]

The strange errors are because the 00_synth.conf file is being read but because it is synths very first build run none of the metadata files exist inside the repo directory so it's not (yet) a valid repo to use. I guess in that situation it isn't falling back to using the FreeBSD repo?

It's not there before "first run", it gets installed on the first install. But the question is why is it being read. Synth tells it to read external repository (meaning don't read anything else). I have to research what could be happening.

 

[marcpearsonti]

Poudriere can keep old versions of package. Take a look at the https://github.com/freebsd/poudriere/blob/master/src/etc/poudriere.conf.sample

# Keep older package repositories. This can be used to rollback a system
# or to bisect issues by changing the repository to one of the older
# versions and reinstalling everything with `pkg upgrade -f`
# ATOMIC_PACKAGE_REPOSITORY is required for this.
# Default: no
#KEEP_OLD_PACKAGES=no
# How many old package repositories to keep with KEEP_OLD_PACKAGES
# Default: 5
#KEEP_OLD_PACKAGES_COUNT=5

 

[marcpearsonti]

If Portmaster can show the versions when showing the "status", Synth should be able to too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg(8) version could be great ! But I'm not a pro !

 

[marcpearsonti]

Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?

 

[marino]

Poudriere can keep old versions of package. Take a look at the https://github.com/freebsd/poudriere/blob/master/src/etc/poudriere.conf.sample

You are not understanding what this does.

It's ALL or NOTHING. You can't roll back "one" package.

Poudriere is making several copies of repositories. If you wanted to roll one thing back, you'd have to roll everything back.

You can do this yourself with net/rsync/sysutils/cpdup. You'd just duplicate the entire repository directory and save it somewhere.

 

[marino]

If portmaster can show the versions when showing the "status", Synth should be able too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg version could be great ! But i'm not a pro !

And what would you do with that information?

 

[marino]

Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?

You are using the wrong terminology.
You asked about packages in a repository. If you change one package and then rebuild the repository, every single package is scanned again and put in the digest. It's the same work for 1 package or 1000 packages.

But now you use words like "packaging" and I don't know what you mean.

Are you talking about packages or repositories? be specific.

 

[kpa]

Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?

This has been beaten to death already many times. There is no way to avoid the recursive pruning and rebuilding of dependent ports even if the end result of the build causes no apparent changes. FreeBSD ports and the package building tools have choice in the matter, it's not a design "feature" but an absolute necessity to make sure all of the potential changes are propagated properly where they have to be propagated. FYI Poudriere does the exact same thing when there are updates to ports.

 

[xtaz]

The "problem" with people like myself who were used to just using ports-mgt/portmaster is that we only ever recompiled the exact ports which had changed and never had a need to redo the dependancies. We trusted that the ABI had not changed at all. The exception is when an entry appeared in /usr/ports/UPDATING telling us that we had to recompile all dependancies, usually when a shared library had its version bumped etc. In general this worked fine and never really caused any issues. But obviously when it comes to a repository you want it to be completely consistent and there's no shortcuts to that. People just need to realise that. One thing I will say is try and do what I have done. Enable package pre-fetching, try and prune the list of ports that have custom options to the bare minimum in /var/db/ports, and properly configure devel/ccache so that it caches object files. These things will greatly speed up the rebuild process.

 

[chrbr]

FreeBSD ports and the package building tools have choice in the matter, it's not a design "feature" but an absolute necessity to make sure all of the potential changes are propagated properly where they have to be propagated.

I have seen that one time since I use devel/poudriere which is just a few weeks. There was a message about an ABI change, too. Therefore it is good to know that the tools can detect this kind of stuff. Otherwise things can creep in and strange effects appear later. I do not understand why it is necessary to rebuild ports to find that out. But I accept that. If rebuild would not be required I am sure that better methods would have been applied in the tool chains by people with higher knowledge than mine.

 

[garry]

xtaz@ if you're correct and /etc/make.conf is considered, that's a problem. It might be a case where I intentionally scan the pkg(8) package (only) using the host ports tree and /etc/make.conf gets sucked in (and I forgot about consequences). I'll have to think about this because that's two reports that stem from that decision.

Three reports actually. That empty string error almost stopped me from continuing when I first started using synth; I fixed it by...

For the life of me I can't remember what I did. If I can retrieve any info I'll add it later. I am an ada programmer! <wink> I wrote "hello world" in ada. I've been reading Feldman and Koffman's "Ada 95: Problem Solving and Program Design".

By the way I hope marino that you can at some point find time to consider the problem with lang/ghc -- Synth always rebuilds it and all of its dependents. After completing a synth just-build ghc and friends look fine but synth rebuild-repository claims "missing dependencies" and throws the freshly built packages away. I've edited the lang/ghc Makefile to add BOOT and BOOTH as normal options along with all other options (rather than options only if bin/ghc and bin/HsColour exist). (and I keep those options turned off because bin/ghc and bin/HsColour will not exist in the chroot(8)). That gets around the problem of Synth complaining always that the options are out of date. My options on lang/ghc are

Options  :
  BCLANG  : on
  BOOT : off
  BOOTH : off
  DOCS  : on
  DYNAMIC  : on
  GCC  : off
  LLVM  : on
  PCLANG  : off
  PROFILE  : on

and it's dependencies are:

> pkg info -d ghc
ghc-7.10.2:
   libiconv-1.14_9
   llvm35-3.5.2_1
   gmp-5.1.3_2
   libffi-3.2.1

Synth builds all 2800 of the ports needed in my repository except lang/ghc.

 

[marino]

garry, if you want to build overnight, this could help:

1. Turn off ncurses mode from configure
2. Exec command env WHYFAIL=1 synth just-build lang/ghc |& tee /tmp/ghc.log (or use script or adjust for whatever shell you have)

If it really does think GHC is always bad, at least we'll know which dependency it doesn't like.

 

[fernandel]

On my system without the option :

====>> Ignoring sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:47:47 CET 2016
build time: 00:00:01

With the option:

===========================================================================
=======================<phase: package  >============================
===>  Building package for fusefs-exfat-1.0.1
===========================================================================
====>> Cleaning up wrkdir
===>  Cleaning for fusefs-exfat-1.0.1
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:48:50 CET 2016
build time: 00:00:04

So I am sure this option should work. IMHO, if the option is written properly then you put it in the wrong file. I do not use ports-mgmt/synth but from what I understood, if you kept the default profile, you should put this option in the /usr/local/etc/synth/LiveSystem-make.conf file.

Thank you very much. Now I have two line in /usr/local/etc/synth/LiveSystem-make.conf:

DEFAULT_VERSIONS+= perl5=5.20 
DISABLE_LICENSES=yes

and it works .

 

[marcpearsonti]

And what would you do with that information?

When managing a production server it's important to know the versions that will be installed. With the exact version change, I can read the changelog (BSD or official site) or refuse to upgrade the ports or prepare some configuration files.

 

[marcpearsonti]

Running synth with WHYFAIL I see these errors about my stalled packages:

rrdtool-1.4.8_9.txz DOCS is OFF but port says it must be ON
cherokee-1.2.104.txz DOCS is ON but port says it must be OFF
databases/rrdtool package has more dependencies than the port requires (8)

I try to delete the ports options with make rmconfig but same errors.

 

[marino]

@marcpearsonti, quarterly or latest? I think databases/rrdtool was already fixed in latest.

 

[marcpearsonti]

marcpearsonti@ quarterly or latest? I think rrdtool was already fixed in latest.

Quarterly ports from 2016Q1 branch

 

[marino]

When managing a production server it's important to know the versions that will be installed. With the exact version change, i can read the changelog (BSD or official site) or refuse to upgrade the ports or prepare some configuration files

You've got a list of changed ports and you've got the port itself (which has the version number). But I'll think about it.