Synth dependencies rebuild overhead ?

ASX · Nov 19, 2016

I'm have observed a a strange behavior for a few times now, and I need to ask about:

just yesterday I completed an update ( synth prepare-system ; pkg upgrade -r Synth), the process built approximatedly 150 pkgs, and installed only very few (4 or 5).

Something similar happened the previous time too. However, once built/installed I have no data to display to check it the whole process is correct or not.

Today, I just updated the port tree ( svnlite upgrade /usr/ports and before doing anything else I checked:

Code:

# pkg info | wc
     704    4812   50498

Code:

# ls /var/synth/live_packages/All/ | wc
     844     844   18858

The difference (844 - 704) easily account for build dependencies, that's fine.

portupgrade report 1 single port needs to be rebuild:

Code:

# portupgrade -a -n
...
   + security/trousers (trousers-0.3.13_1 -> trousers-0.3.14)
--->  Packages processed: 1 done, 10 ignored, 0 skipped and 0 failed

(The 10 ignored are out of port tree pkgs)

synth status report 113 pkgs need to be rebuilt:

Code:

# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of x11-themes/ghostbsd-wallpapers failed (port deleted), it will not be considered.
Scan of x11/xfce-installed-settings failed (port deleted), it will not be considered.
Scan of sysutils/inxi failed (port deleted), it will not be considered.
Scan of net-mgmt/networkmgr failed (port deleted), it will not be considered.
Scan of x11/ghostbsd-slim-theme failed (port deleted), it will not be considered.
Scan of x11-themes/ghostbsd-icons failed (port deleted), it will not be considered.
Scan of x11/ghostbsd-installed-common-settings failed (port deleted), it will not be considered.
Scan of devel/ssft failed (port deleted), it will not be considered.
Scan of x11-themes/ghostbsd-mate-themes failed (port deleted), it will not be considered.
Scan of sysutils/ghostbsd-grub2-settings failed (port deleted), it will not be considered.
Scanning existing packages.
gnutls-3.4.16.txz failed dependency check.
cups-2.2.1.txz failed dependency check.
gtk2-2.24.29_2.txz failed dependency check.
gtk3-3.18.8_3.txz failed dependency check.
xfce4-conf-4.12.1.txz failed dependency check.
libxfce4menu-4.12.1_1.txz failed dependency check.
libwnck-2.30.7.txz failed dependency check.
garcon-0.4.0_1.txz failed dependency check.
libexo-0.10.7.txz failed dependency check.
xfce4-panel-4.12.1.txz failed dependency check.
ghostscript9-agpl-base-9.16_5.txz failed dependency check.
glib-networking-2.46.1_1.txz failed dependency check.
libsoup-2.52.2.txz failed dependency check.
texlive-base-20150521_13.txz failed dependency check.
texlive-texmf-20150523_3.txz failed dependency check.
geoclue-2.3.0.txz failed dependency check.
gconf2-3.2.6_4.txz failed dependency check.
libsoup-gnome-2.52.2.txz failed dependency check.
tex-formats-20150521_2.txz failed dependency check.
rest-0.7.93.txz failed dependency check.
webkit2-gtk3-2.8.5_6.txz failed dependency check.
gcr-3.18.0.txz failed dependency check.
libglade2-2.6.4_8.txz failed dependency check.
qt5-printsupport-5.6.2.txz failed dependency check.
tex-xmltex-1.9_2.txz failed dependency check.
policykit-gnome-0.9.2_7.txz failed dependency check.
gnome-online-accounts-3.18.4_1.txz failed dependency check.
uhttpmock-0.5.0.txz failed dependency check.
libcanberra-0.30_3.txz failed dependency check.
libgdata-0.17.4.txz failed dependency check.
tex-jadetex-3.13_3.txz failed dependency check.
qt5-webkit-5.6.2.txz failed dependency check.
gnome-mount-0.8_12.txz failed dependency check.
ffmpeg-2.8.8_5,1.txz failed dependency check.
gvfs-1.26.3_2.txz failed dependency check.
qt5-assistant-5.6.2.txz failed dependency check.
tex-dvipsk-5.995_1.txz failed dependency check.
py27-gtk2-2.24.0_4.txz failed dependency check.
docbook-utils-0.6.14_13.txz failed dependency check.
Thunar-1.6.10_2.txz failed dependency check.
doxygen-1.8.12,2.txz failed dependency check.
qt5-designer-5.6.2.txz failed dependency check.
webkit-gtk2-2.4.11_4.txz failed dependency check.
claws-mail-3.14.1.txz failed dependency check.
gtkmm24-2.24.4_2.txz failed dependency check.
gtksourceview2-2.10.5_4.txz failed dependency check.
unique-1.1.6_6.txz failed dependency check.
vte3-0.42.4_1.txz failed dependency check.
gnupg-2.1.15.txz failed dependency check.
orage-4.12.1_1.txz failed dependency check.
xfce4-notifyd-0.3.4.txz failed dependency check.
qt5-linguist-5.6.2.txz failed dependency check.
cups-pk-helper-0.2.5_1.txz failed dependency check.
libspectre-0.2.8.txz failed dependency check.
py27-pycups-1.9.73_1.txz failed dependency check.
firefox-50.0_2,1.txz failed dependency check.
thunderbird-45.4.0_2.txz failed dependency check.
xfce4-settings-4.12.1.txz failed dependency check.
samba36-3.6.25_3.txz failed dependency check.
libxfce4gui-4.10.0_5.txz failed dependency check.
xfce4-appfinder-4.12.0.txz failed dependency check.
xfce4-desktop-4.12.3.txz failed dependency check.
xfce4-session-4.12.1_3.txz failed dependency check.
xfce4-wm-4.12.3.txz failed dependency check.
gtk-xfce-engine-3.2.0.txz failed dependency check.
mousepad-0.4.0_2.txz failed dependency check.
pavucontrol-3.0.txz failed dependency check.
pinentry-gnome3-0.9.7.txz failed dependency check.
keybinder-0.3.1.txz failed dependency check.
keybinder-gtk3-0.3.1.txz failed dependency check.
xfce4-terminal-0.8.1.txz failed dependency check.
xfce4-volumed-pulse-0.2.2.txz failed dependency check.
xfce4-xkb-plugin-0.7.1.txz failed dependency check.
virtualbox-ose-5.1.8.txz failed dependency check.
cups-filters-1.11.4.txz failed dependency check.
cups-smb-backend-1.0_7.txz failed dependency check.
ghostscript9-agpl-x11-9.16_2.txz failed dependency check.
system-config-printer-1.4.7_3.txz failed dependency check.
firefox-i18n-50.0.txz failed dependency check.
py27-webkitgtk-1.1.8_6.txz failed dependency check.
xfce4-smartbookmark-plugin-0.4.6_1.txz failed dependency check.
claws-mail-fancy-3.14.1.txz failed dependency check.
claws-mail-pdf_viewer-3.14.1.txz failed dependency check.
thunderbird-i18n-45.4.0.txz failed dependency check.
xfburn-0.5.4_4.txz failed dependency check.
xfce4-battery-plugin-1.0.5_4.txz failed dependency check.
xfce4-genmon-plugin-3.4.0_5.txz failed dependency check.
xfce4-power-manager-1.6.0.txz failed dependency check.
xfce4-wavelan-plugin-0.5.12_1.txz failed dependency check.
xfce-4.12_1.txz failed dependency check.
mplayer-1.3.0.20160912_1.txz failed dependency check.
evince-lite-3.18.2_1.txz failed dependency check.
gpicview-0.2.5.txz failed dependency check.
ristretto-0.8.1.txz failed dependency check.
shotwell-0.24.2.txz failed dependency check.
hexchat-2.12.3.txz failed dependency check.
xarchiver-0.5.4.7.txz failed dependency check.
xfce4-dict-plugin-0.7.2.txz failed dependency check.
libreoffice-5.2.3_2.txz failed dependency check.
vim-8.0.0082.txz failed dependency check.
xfce4-datetime-plugin-0.6.2_4.txz failed dependency check.
xfce4-timer-plugin-1.6.0_1.txz failed dependency check.
xfce4-mixer-4.11.0_3.txz failed dependency check.
xfce4-pulseaudio-plugin-0.2.4.txz failed dependency check.
gnome-keyring-3.18.3_1.txz failed dependency check.
nvidia-settings-355.11_3.txz failed dependency check.
trayer-1.1.6.txz failed dependency check.
xfce4-quicklauncher-plugin-1.9.4_17.txz failed dependency check.
xfce4-screenshooter-plugin-1.8.2_2.txz failed dependency check.
xfce4-taskmanager-1.1.0_1.txz failed dependency check.
xfce4-verve-plugin-1.1.0_1.txz failed dependency check.
zenity-3.18.0.txz failed dependency check.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  U => security/trousers (0.3.13_1 => 0.3.14)
  R => security/gnutls
  R => print/cups
  R => x11-toolkits/gtk20
  R => x11-toolkits/gtk30
  R => x11/xfce4-conf
  R => x11/libxfce4menu
  R => x11-toolkits/libwnck
  R => sysutils/garcon
  R => x11/libexo
  R => x11-wm/xfce4-panel
  R => print/ghostscript9-agpl-base
  R => net/glib-networking
  R => devel/libsoup
  R => print/texlive-base
  R => print/texlive-texmf
  R => net/geoclue
  R => devel/gconf2
  R => devel/libsoup-gnome
  R => print/tex-formats
  R => devel/librest
  R => www/webkit2-gtk3
  R => security/gcr
  R => devel/libglade2
  R => print/qt5-printsupport
  R => print/tex-xmltex
  R => sysutils/policykit-gnome
  R => net/gnome-online-accounts
  R => net/uhttpmock
  R => audio/libcanberra
  R => devel/libgdata
  R => print/tex-jadetex
  R => www/webkit-qt5
  R => sysutils/gnome-mount
  R => multimedia/ffmpeg
  R => devel/gvfs
  R => devel/qt5-assistant
  R => print/tex-dvipsk
  R => x11-toolkits/py-gtk2
  R => textproc/docbook-utils
  R => x11-fm/thunar
  R => devel/doxygen
  R => devel/qt5-designer
  R => www/webkit-gtk2
  R => mail/claws-mail
  R => x11-toolkits/gtkmm24
  R => x11-toolkits/gtksourceview2
  R => x11-toolkits/unique
  R => x11-toolkits/vte3
  R => security/gnupg
  R => deskutils/orage
  R => deskutils/xfce4-notifyd
  R => devel/qt5-linguist
  R => print/cups-pk-helper
  R => print/libspectre
  R => print/py-pycups
  R => www/firefox
  R => mail/thunderbird
  R => sysutils/xfce4-settings
  R => net/samba36
  R => x11-toolkits/libxfce4gui
  R => misc/xfce4-appfinder
  R => x11-wm/xfce4-desktop
  R => x11-wm/xfce4-session
  R => x11-wm/xfce4-wm
  R => x11-themes/gtk-xfce-engine
  R => editors/mousepad
  R => audio/pavucontrol
  R => security/pinentry-gnome3
  R => x11/keybinder
  R => x11/keybinder-gtk3
  R => x11/xfce4-terminal
  R => deskutils/xfce4-volumed-pulse
  R => deskutils/xfce4-xkb-plugin
  R => emulators/virtualbox-ose
  R => print/cups-filters
  R => print/cups-smb-backend
  R => print/ghostscript9-agpl-x11
  R => print/system-config-printer
  R => www/firefox-i18n
  R => www/py-webkitgtk
  R => www/xfce4-smartbookmark-plugin
  R => mail/claws-mail-fancy
  R => mail/claws-mail-pdf_viewer
  R => mail/thunderbird-i18n
  R => sysutils/xfburn
  R => sysutils/xfce4-battery-plugin
  R => sysutils/xfce4-genmon-plugin
  R => sysutils/xfce4-power-manager
  R => sysutils/xfce4-wavelan-plugin
  R => x11-wm/xfce4
  R => multimedia/mplayer
  R => graphics/evince-lite
  R => graphics/gpicview
  R => graphics/ristretto
  R => graphics/shotwell
  R => irc/hexchat
  R => archivers/xarchiver
  R => textproc/xfce4-dict-plugin
  R => editors/libreoffice
  R => editors/vim
  R => x11-clocks/xfce4-datetime-plugin
  R => x11-clocks/xfce4-timer-plugin
  R => audio/xfce4-mixer
  R => audio/xfce4-pulseaudio-plugin
  R => security/gnome-keyring
  R => x11/nvidia-settings
  R => x11/trayer
  R => x11/xfce4-quicklauncher-plugin
  R => x11/xfce4-screenshooter-plugin
  R => x11/xfce4-taskmanager
  R => x11/xfce4-verve-plugin
  R => x11/zenity
Total packages that would be built: 113
The complete build list can also be found at:
/tmp/synth_status_results.txt
[/U]

The question is: what's is triggering all those "failed dependency check" on an otherwise updated system ? Is something wrong in my setup or am I doing something wrong ?

Must be noted that I have no issue, other than some (apparently) wasted time rebuilding those packages.

I have not performed further steps right now, if needed I can provide info about the current state of the things.

PacketMan · Nov 19, 2016

I get that often on my GNOME3 machine. In fact it was all up to date a couple days ago. But I decided to run an upgrade again to install a new port, and blam a ton of fail dependency checks. Through other discussion I have been told that is normal expected behavior. Now I am trying to figure out with synth aborting on www/webkit2-gtk3. (See other topic I posted earlier today.) Fun times!

marino · Nov 19, 2016

it is normal behavior.
I've been thinking about an option that would use multiple passes to lessen this effect, but as you might imagine, it increases the complexity greatly.

ASX · Nov 19, 2016

marino@ said:
it is normal behavior.
I've been thinking about an option that would use multiple passes to lessen this effect, but as you might imagine, it increases the complexity greatly.

OK, I'm fine with that, just needed to be sure I was not missing something.

marino · Nov 19, 2016

pkg(8) just checks for version changes. poudriere and synth aggressively rebuild (or most conservatively) even when there was no version change because the dependency chain changed somewhere.

kpa · Nov 19, 2016

There have been earlier threads on the same issue but I couldn't find them now.

Basically what happens is that it's impossible to know in advance if a rebuild of a port would trigger a need to rebuild one or more of the dependent ports of the changed port. That knowledge if it was available would save your from rebuilding all of the dependents (and their dependents recursively) if it was available but we don't have anything resembling a universal ABI compatibility checker that could take two ports and be able to decide if the two will operate together correctly if installed at the same time. If you try to be cheap and not rebuild the dependents by force you're going to come across a dependent port that should have been rebuilt and the only way you can find that out is running the software and find out that something went wrong. This is not acceptable on a build cluster that is supposed to build ports to packages without manual intervention. This is how the early versions of ports-mgmt/poudriere behaved and it produced broken builds where the cause of the breakage was really hard to figure out and poudriere had to be changed to rebuild all of the dependents recursively like it does now.

This is not a FreeBSD specific problem. Any system that uses dynamic linking in the same way as FreeBSD does is going face the same difficulty in determining if a change in one binary or dynamic link library is going to trigger the same rebuild of the dependents.

ASX · Nov 19, 2016

kpa,
thanks for the additional explanation.
If that is the case I would expect portupgrade producing the same behavior, which is not what I see.

Now, I'm possibly more confused, there is a post where marino@ state it is possible to lessen the build but complex/difficult, there is yours where you state it is near to "impossible", and there is the FreeBSD handbook which happily suggest to rely on portmaster or portupgrade.
handbook: https://www.freebsd.org/doc/handbook/ports-using.html

There is still a portion of your explanation that is difficult to accept:

kpa said:
If you try to be cheap and not rebuild the dependents by force you're going to come across a dependent port that should have been rebuilt and the only way you can find that out is running the software and find out that something went wrong.

... (unless you meant to write "building the software" instead of "running the software")

The fact is that, even if those dependent packages are rebuilt, they are not going to be installed and I have not noticed issues. (however, I'm using ports/synth from just few weeks ... may be just I'm lucky ?)

If I have to trust your assertion above, I must conclude that using (in my case) pkg upgrade -r Synth is not enough to have all the required pkgs updated, instead I should force some reinstall of all the rebuilt packages.

However I look at the whole, something is not right, one or more of these:

- misleading handbook: because it suggest to use portupgrade / portmaster knowing their job is not enough.
- build overhead, because of difficult in tracking dependency relations
- missing reinstallation, because pkgs are built but later not installed
- me, not forcing the pkgs reinstall, just upgrading from local rebuilt repository.

which one(s) ?

Remington · Nov 19, 2016

You might want to look into adding devel/ccache to cut down the time on rebuilding packages that hasn't been changed.

kpa · Nov 19, 2016

I absolutely meant "running the software". ABI compatibility issues may not manifest at compile time at all but instead they will show up at run time as crashes and other problems *). Build time problems are much nicer because you have at least a clue based on the error messages where to start looking at. In case of runtime crashes you have to first load the program into a debugger and then start looking where the problem might be.

*) In fact FreeBSD has had some problems maintaining it's promised stable ABI in the base system where a unintended change has caused base system programs start to crash even if they compile fine.

ASX · Nov 19, 2016

Remington thanks, I'm already using ccache.
kpa, fine ... then should I force the re-installation of those rebuilt packages ?

marino · Nov 19, 2016

ASX said:
kpa,
thanks for the additional explanation.
If that is the case I would expect portupgrade producing the same behavior, which is not what I see.

Why would you expect that? It's not logical to do so.

portupgrade uses the same garbage foundation as portmaster.
You should *never* compare portupgrade to poudriere or synth, they are apples to oranges. If you must compare it to something, limit those comparisons to portmaster.

kpa · Nov 19, 2016

ASX said:
kpa, fine ... then should I force the re-installation of those rebuilt packages ?

No, they are still the same packages for pkg(8) because in its view the packages haven't changed even if they have been recompiled. You can trust the there is no need to reinstall anything that pkg doesn't see as an update because this forced recompilation of the dependent ports has already taken care of figuring out if something had to be rebuilt after all. This of course assuming that you're building your ports using Poudriere or Synth, the other tools are not to be trusted in this manner at all.

ASX · Nov 19, 2016

marino@ said:
Why would you expect that? It's not logical to do so.

portupgrade uses the same garbage foundation as portmaster.

That's what is officially suggested in the handbook to upgrade / install from ports.
That both are based on garbage is something I don't know, until someone like you tell me that.

kpa said:
No, they are still the same packages for pkg(8) because in its view the packages haven't changed even if they have been recompiled. You can trust the there is no need to reinstall anything that pkg doesn't see as an update

Then I don't follow your reasoning about crashes happening after some component update. I can only think those packages are rebuilt for nothing, because in the end aren't going to be installed, and accept that like normal behavior. Like said above I'm fine with that.

kpa · Nov 19, 2016

ASX said:
That's what is officially suggested in the handbook to upgrade / install from ports.
That both are based on garbage is something I don't know, until someone like you tell me that.

Then I don't follow your reasoning about crashes happening after some component update. I can only think those packages are rebuilt for nothing, because in the end aren't going to be installed, and accept that like normal behavior. Like said above I'm fine with that.

You have read what I wrote as what would happen if the package builders like Poudriere and Synth were not doing the "excessive" rebuild. They are not rebuilt for nothing, there is always a possiblity that one or more of the rebuilds trigger a change in the shared library dependencies and that's something that pkg(8) will notice and you will be then presented with a changed package for that dependent port.

ASX · Nov 19, 2016

kpa, I will make it a little more practical here:
portupgrade -a-n show that 1 pkg need to be rebuilt: security/trousers
synth status report the same need to be rebuilt, from my initial report :

ASX said:
U => security/trousers (0.3.13_1 => 0.3.14)

in additions synth status report some more 100 ports to be rebuild all marked with an R in example:

ASX said:
R => editors/libreoffice

As far as I can see, editors/libreoffice depend on security/trousers, not the inverse.
ports-mgmt/synth rebuild editors/libreoffice ... libreoffice version is not changed anyway, is renewed in synth local repository and later is not going to be installed.

How is libreoffice is not going to be rebuilt for nothing ? How would pkg notice the package is newer and need to be reinstalled if the libreoffice version has not been changed ?

marino · Nov 19, 2016

ASX said:
That's what is officially suggested in the handbook to upgrade / install from ports.

It's not. These are no longer recommended. They are mentioned as "tools that exist". They used to be recommended but the words about recommendation should have been removed. If you can point to explicit "recommendations" of either portmaster or portupgrade, please paste the reference here so we can get those adjusted.

ASX · Nov 19, 2016

marino@ said:
It's not. These are no longer recommended. They are mentioned as "tools that exist". They used to be recommended but the words about recommendation should have been removed. If you can point to explicit "recommendations" of either portmaster or portupgrade, please paste the reference here so we can get those adjusted.

I started by using pkgs only, and later went to use ports, thus I went to "upgrade ports":

https://www.freebsd.org/doc/handbook/ports-using.html

4.5.3. Upgrading Ports
..."
To perform the actual upgrade, use either Portmaster or Portupgrade.

It is not "recommended" ... but that is what I read ... I have also written "officially suggested", not "recommended".

marino · Nov 19, 2016

yes, that line should be changed or removed.

ASX · Nov 19, 2016

well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.

marino · Nov 19, 2016

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214679

marino · Nov 19, 2016

ASX said:
well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.

I tried to get portmaster marked as deprecated with no expiration date because it was unmaintained and people went ape-****. Finally somebody put their name on it, but their next commit to it will be their first. It's not maintained. Even if it did perfectly what it was designed it do, it's defective by design. It's like how Linux thinks about svn. (There's no way to do cvs "right"). There's no way to make portmaster right.

wblock@ · Nov 20, 2016

ASX said:
well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.

Opinions vary.

ports-mgmt/portmaster and ports-mgmt/portupgrade still work. ports-mgmt/portmaster is nice because it does not need Ruby and the defaults are somewhat more in line with preferred procedure. I think the separate port database requirement of ports-mgmt/portupgrade has been fixed with pkg, but I used that for years and see no reason to go back to it.

ASX · Nov 20, 2016

wblock@
there was an implicit cross-thread reference in my previous post, addressing new ports requirements, where I had a little discussion about requirements not being listed in handbook/documentation: It was (partially) supposed to be ironic.
https://forums.freebsd.org/threads/58578/#post-335068

wblock@ said:
ports-mgmt/portmaster is nice because it does not need Ruby

Please correct me if I'm wrong, I tried portmaster very briefly and if I remember correctly it will install the requested/upgraded port along with the build dependencies. (in addition to run-dependencies).

Do people using portmaster really need to worry about Ruby considering the context ?

Beside that I'm really interested in the next reply from kpa, his comments about possible crashes. it is a key factor for me to take a final position.

marino · Nov 20, 2016

ports-mgmt/portmaster and ports-mgmt/portupgrade still work

If you play fast and loose with the definition of "work" ...

Among people that develop ports, opinions do not really vary.

wblock@ · Nov 20, 2016

ASX said:
Do people using portmaster really need to worry about Ruby considering the context?

portmaster does not need or use Ruby, so that does not affect its use. portupgrade does use Ruby, so upgrades can be a problem.

I'm not sure what you mean about the other thread.

There is a section on Poudriere in the Handbook. I did not write it, but spent several hours working with the author on it and committed it.

Synth dependencies rebuild overhead ?

ASX

Guest

PacketMan

marino

ASX

Guest

marino

kpa

ASX

Guest

Remington

kpa

ASX

Guest

marino

kpa

ASX

Guest

kpa

ASX

Guest

marino

ASX

Guest

marino

ASX

Guest

marino

marino

wblock@

ASX

Guest

marino

wblock@