Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

marino@, is it really better to use the quarterly branch. I use this because i was thinking that ports receive only security updates but after looking to the svnweb site some ports has been upgraded too. So what's the real reason FreeBSD has created these ports branches ?
 
Is there a way to delete a package file from the repository directory (other than rm <package>.txz)?

My repository has packages for MySQL client & server for both v5.5 and v5.6 -- one of my ports had been compiled against 5.5, and Synth successfully rebuilt it against 5.6, but 5.5 was still built as part of the initial population of the repository. So nothing uses 5.5, and I don't see a need to keep it in the repository (especially since the default in ports is 5.6 anyway).

I can imagine there may be other package files in the repository directory that will be "orphaned" in the future if they are no longer a build or run dependency following some future upgrade that changes the dependent port's dependency lists.

The only sort-of-analogous command I found in Poudriere would be poudriere pkgclean, but that allows you to list package files you want to keep, instead of files you want to delete.
 
That's the only way, but it doesn't really matter if you have extra packages in a repo that aren't installed on the system.
 
I've found an odd one this morning. I have two ports which have a local patch file in the files/ directory that I have written. If I install the port manually then the patch is applied and works. If I look at the package that synth made then the patch has not been applied. Not sure what could cause this?
 
I don't see how it's possible. The ports directory is mounted read-only. The patch would be available to the port.
You can use synth test with the ENTERAFTER env var (see man page) to jump into the build and verify for sure whether or not the patch was applied.
 
Yep. Ignore me. I've obviously done something silly. The patch is in fact *not* being applied if I install the port manually. It was working last week so I have no idea why it's not working now as the port hasn't been updated at all. I'll investigate.
 
marino@ said:
garry, this could help:
1) turn off ncurses mode from configure
2) exec command env WHYFAIL=1 synth .....

If it really does think GHC is always bad, at least we'll know which dependency it doesn't like.
Click to expand...

I built an up-to-date repository with Synth, built lang/ghc, verified that the package was in the packages directory, and then tried to rebuild the repository with Synth. It detected a "missing dependency" in lang/ghc and removed the "bad" package. The result is that lang/ghc can not be built and added to the repository. The details are mystifying:
Code: 
synth prepare-system
env WHYFAIL=1 synth rebuild-repository
The result is that lang/ghc fails the dependency check:
Code: 
Scanning entire ports tree.
lang/ghc package has less dependencies than the port requires (5)
Query: converters/libiconv:libiconv-1.14_9
devel/llvm35:llvm35-3.5.2_1
math/gmp:gmp-5.1.3_3
devel/libffi:libffi-3.2.1
                                                                                    <<<blank line?>>>
ghc-7.10.2.txz failed dependency check.
Local repository successfully rebuilt
But libiconv-1.14_9, llvm35-3.5.2_1, gmp-5.1.3_3, and libffi-3.2.1 are right there in the repository. No dependency appears to be missing. I took a look at the +COMPACT-MANIFEST for the GHC package that Synth built (before Synth threw it away) and didn't see anything of note. Is the blank line in the output significant? Could that be a "dependency" on non-printing chars? There are not 5 dependencies listed.

I'm at a loss. Please let me know how further to debug this.
 
You are not interpreting the error message correctly.
The port Makefile says there are 5 dependencies, but the package only lists 4 dependencies.

This is almost always a problem in the port Makefile.
 
It would help to see the portion of the Synth log that says what options were set before the build.
 
Do you know why pkg(8) installs some things from the FreeBSD repo and some things from the Synth repo? If I do a pkg upgrade -f then it lists things like this:

Code: 
  gpgme-1.6.0 [FreeBSD]
  gnupg-2.1.8 [Synth]
  fdk-aac-0.1.4 [FreeBSD]
  expat-2.1.0_3 [FreeBSD]
  en-aspell-7.1.0_1 [Synth]
  duplicity-0.6.25 [Synth]

The packages for the FreeBSD ones are all present and correct in /var/synth/live_packages/All. I guess if the packages are identical in both repos it maybe just picks one at random?
 
Synth should have priority. If I had to guess, pkg(8) goes to the repo where the last package is from.
You can probably force everything to synth if you "pkg prime-list > mylist", the delete all packages, then use the prime-list to reinstall (all from Synth)
 
Thanks guys. I disabled the FreeBSD repo and then ran the upgrade again. After then reenabling the repo and running the command once more it shows Synth for everything. Makes sense now after reading about CONSERVATIVE_UPGRADE.
 
Let's try this again. If no new problems reveal with version 0.99_7, I'll make the v1.00 release.
Code: 
ports-mgmt/synth: Yet another release candidate

Unfortunately, there's been a bit too much change since 0.99_6 to
confidently release version 1.00, so another release candidate is
necessary.  Both new features and bug fixes were added.

New features:
  * Provide ability to define environment variables in a profile
    (/usr/local/etc/synth/<profile>-environment)
  * Support fetching by proxy using these environment variables
  * Add zsh and bash completion scripts
  * Accept port origins with trailing file separators (so people
    using completion scripts don't have to backtrack to remove them)
  * In text (non-curses) mode, output the current package build
    tally every 200 seconds (approximately)

Bug fixes:
  * Fix support for system roots that don't match host (e.g.
    ARCH, OSRELEASE, OSVERSION, etc
  * Fix ABI check for system roots that don't match host
  * Remove effect of system /etc/make.conf (originally seen when
    MAKE_JOBS_NUMBER was defined there and disabled synth)
 
garry, if I had to bet money, I think Line 83 is the culprit,
Code: 
LIB_DEPENDS+=           libutil.so.9:${PORTSDIR}/misc/compat9x
My guess is you can comment that out and it won't rebuild anymore.
 
On my system Synth keeps rebuilding devel/libc++ because it fails dependency. What does it mean? I don't have libc++ package installed:

Code: 
root@bclinton:libc++# pkg info libc++
pkg: No package(s) matching libc++

My system:

Code: 
root@bclinton:libc++# uname -a
FreeBSD bclinton 10.3-BETA2 FreeBSD 10.3-BETA2 #0 r295581: Fri Feb 12 13:54:41 PST 2016     toor@bclinton:/usr/obj/usr/src/sys/GENERIC  amd64

Code: 
root@bclinton:libc++# pkg info synth
synth-0.99_7
...

Code: 
root@bclinton:synth# tail devel___libc++.log
========================< phase : package         >========================
===>  Building package for libc++-208080
actual-package-depends: dependency on /usr/lib/libcxxrt.so not registered (normal if it belongs to base)
file sizes/checksums   [237]: ... done
packing files          [237]: ... done
packing directories      [0]: . done
===========================================================================

Finished: Friday, 12 FEB 2016 at 00:00:39 UTC
Duration: 00:00:10

Thanks a lot for creating such a great program!
 
marino@ said:
garry@, if I had to bet money, I think Line 83 is the culprit,
Code: 
LIB_DEPENDS+=           libutil.so.9:${PORTSDIR}/misc/compat9x
My guess is you can comment that out and it won't rebuild anymore.
Click to expand...

You would not lose your money. After writing my last post and seeing your reply I took a closer look in the Makefile for the "missing" depends. I did just as you suggest; I removed the compat9x depends line (libutil.so.9 is provided by the system -- I'm on FreeBSD 11 head). That solved the problem. Thank you, now it's time to cook myself a Friday night fried fish dinner and enjoy a movie! After the movie I'll let Synth build my entire repository and I'll stop using poudriere.

One question: is the misc/compat9x dependency statement a bug in the lang/ghc Makefile? (portlint(1) gives many other complaints about that port). Should I report a bug against lang/ghc? Or is this just a mismatch between how Synth scans Makefiles and what this one port does? I checked another port that may require misc/compat9x, archivers/rar, but that works ok because the misc/compat9x port is pulled in by that package.
 
garry there is already a PR filed against that port for the same reason. I found that out after I contacted the maintainers. Since libutil doesn't appear to even be provided by misc/compat9x, I think it's a bug in any sense of the word. I predict it will be removed once my finding is confirmed.
 
tankist02 said:
On my system Synth keeps rebuilding devel/libc++ because it fails dependency. What does it mean? I don't have libc++ package installed:
Click to expand...

You're not understanding what's it doing. You would *never* check the host system in response to a synth error. It brings in it's own packages that it already built, it would never rely on what's installed in the host.

It's probably rebuilding for the same reason lang/ghc (discussed above) and others before that rebuild: An issue with the Makefile that causes the dependency specifications to be different than what the built package actually has. In the case of GHC, the Makefile said there were 5 dependencies but the scanned package only had 4 so Synth declared that it failed the dependency check. I'm guessing the same thing is happening here.

Synth has been finding bugs in the port system due to this strictness.
 
@marino, I am wondering if I am having an issue. I've decided to try synth on a FreeBSD 10.2-RELEASE machine running GNOME3. The short story is Synth is continuing having to build a pile of ports. 1st time after the initial go-around it was 500+, then 300+ now 200+. Also its skipping some and ignoring 1. I watch it finish up last night and it look like it actually only installed 6 or so of the newly built packages meaning only 6 or ports were actually upgraded. With ports like www/webkit-gtk3 being rebuilt every time I have to ask why. NOTE: pretty sure all the ports being rebuilt are failing dependency check. When I used Portmaster I only ever had one missing library issue, never anything like this. These rebuilds are over 48 hours long. So by that time the ports tree has been updated, so I do my portsnap fetch update and presto - Synth says 200+ need to be rebuilt. So my question is why? What log can I look at to determine why? Please note ports like www/webkit-gtk3, www/firefox and editors/openoffice-4 are usually always rebuilt. OpenOffice was last updated Jan 29th, and Firefox was updated 9 Feb. Both of these were rebuilt a few days ago, and yet synth is telling me its going to do them again.

I am gonna do more testing; including issuing the upgrade command immediately after Synth finishes, and after a reboot, but both before I do another portsnap fetch update.

I know you have been finding issues with lots of ports and thus maybe that is the root cause on this GNOME3 machine. Thoughts?
 
... newly built packages meaning only 6 or ports were actually upgraded.
Click to expand...
This is not bad news. pkg(8) only replaces what clearly is obsolete. Synth (and Poudriere) rebuild even when a LIB/RUN dependency fails, no matter how low. When they rebuild, pkg(8) might not see a difference. You could argue, "but then why did I have to rebuild?" Because you can't know beforehand and even pkg(8) perhaps needs to update but it can't detect that it does.

I've said it before, but LibreOffice and KDE4 are basically broken every single day. They have so many dependencies that each have dependencies which in turn has dependencies (and so on) that the chance of missing a cascading failure any given day is low, and that's PER DAY. You should basically expect it.

Now that doesn't mean there isn't a bug in a port Makefile.
You can only tell if you build everything and then without updating the ports tree, build everything again. Ideally Synth will say "nothing to do". If it doesn't, that means it deleted a port a started a cascade. You need to log what it's doing (perhaps in text mode) to determine which port gets deleted first. Having WHYFAIL defined in environment will help diagnose as well (see GHC above)
 
Note that quarterly branches are much more stable so things are less likely to need rebuilding between runs.
 
One more thing on pkg(8). This is why it's important for ports devs to bump the PORTREVISION. If that pkg name (including version/port revision) don't change, then pkg will not reinstall the package, even if it's different, because it doesn't know that it's different.
 
You must log in or register to reply here.
Back
Top