portmaster doesn't fetch build dependencies

[kpect]

synth(1):

     <profile>-make.conf
                       This is an optional, user-provided file. If it exists,
                       the builder's /etc/make.conf will be appended with the
                       contents of this file. For the default profile, the
                       file would normally be located at
                       /usr/local/etc/synth/LiveSystem-make.conf

According to the man, /etc/make.conf is appended with <profile>-make.conf so it doesn't matter where "OPTIONS_UNSET=..." is located, right? So it can be processed anywhere, or not?

 

[ASX]

According to the man, /etc/make.conf is appended with <profile>-make.conf so it doesn't matter where "OPTIONS_UNSET=..." is located, right? So it can be processed anywhere, or not?

No, it say: "the builder's /etc/make.conf", which is not your system's /etc/make.conf.

Place your options in <profile>-make.conf.
If you want to check early, the merged options are logged in each build log file.

 

[kpect]

Thanx ASX, I'll check that.

 

[kpect]

Phew!

# portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
...
/usr/ports/net/luasocket/
/usr/ports/net/ocserv/
/usr/ports/net/pecl-smbclient/

# pkg version -vIL=
ffmpeg-3.2.3,1                     <   needs updating (index has 3.2.4,1)

$ doas synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
zip-3.0_1.txz failed option check.
libogg-1.3.2_2,4.txz failed option check.
check-0.11.0.txz failed option check.
libvorbis-1.3.5_1,3.txz failed option check.
bison-3.0.4,1.txz failed option check.
binutils-2.27_5,1.txz failed option check.
ncurses-6.0_3.txz failed option check.
cmake-3.7.2.txz failed option check.
ninja-1.7.2,2.txz failed option check.
Scanning existing packages.
libdrm-2.4.75,1.txz failed option check.
help2man-1.47.4.txz failed option check.
giflib-5.1.4.txz failed option check.
yasm-1.3.0.txz failed option check.
gmake-4.2.1_1.txz failed option check.
adacurses-20150808_4.txz failed option check.
bash-4.4.12.txz failed option check.
python27-2.7.13_1.txz failed option check.
libgpg-error-1.26.txz failed option check.
nasm-2.11.08_1,1.txz failed option check.
jpeg-turbo-1.5.1.txz failed option check.
m4-1.4.18,1.txz failed option check.
libXpm-3.5.12.txz failed option check.
readline-6.3.8.txz failed option check.
texinfo-6.1.20160425,1.txz failed option check.
libtorrent-0.13.6_1.txz failed option check.
chrony-3.0.txz failed option check.
mksh-54.txz failed option check.
ini_file_manager-03_2.txz failed option check.
cdrtools-3.01.txz failed option check.
libgcrypt-1.7.6.txz failed option check.
libxcb-1.12.txz failed option check.
gtk2-2.24.29_3.txz failed option check.
dejavu-2.37.txz failed option check.
fontconfig-2.12.1,1.txz failed option check.
autoconf-2.69_1.txz failed dependency check.
automake-1.15_1.txz failed dependency check.
libtool-2.4.6.txz failed dependency check.
libxslt-1.1.29_1.txz failed dependency check.
xcb-proto-1.12.txz failed dependency check.
libX11-1.6.4,1.txz failed dependency check.
libXext-1.3.3_1,1.txz failed dependency check.
python2-2_3.txz failed dependency check.
libXfixes-5.0.3.txz failed dependency check.
py27-setuptools27-32.1.0.txz failed dependency check.
scons-2.5.1.txz failed dependency check.
libXt-1.1.5,1.txz failed dependency check.
libXdamage-1.1.4_3.txz failed dependency check.
libXrender-0.9.10.txz failed dependency check.
libXv-1.0.11,1.txz failed dependency check.
libXvMC-1.0.10.txz failed dependency check.
libglapi-13.0.4.txz failed dependency check.
libXi-1.7.9,1.txz failed dependency check.
glib-2.46.2_4.txz failed dependency check.
libXxf86vm-1.1.4_1.txz failed dependency check.
libxkbfile-1.0.9.txz failed dependency check.
libXmu-1.1.2_3,1.txz failed dependency check.
mpfr-3.1.5_1.txz failed dependency check.
xcb-util-0.4.0_2,1.txz failed dependency check.
libGL-13.0.4_1.txz failed dependency check.
xcb-util-renderutil-0.3.9_1.txz failed dependency check.
cairo-1.14.6_1,2.txz failed dependency check.
gbm-13.0.4.txz failed dependency check.
libXinerama-1.1.3_3,1.txz failed dependency check.
libXrandr-1.5.1.txz failed dependency check.
xkbcomp-1.3.1.txz failed dependency check.
libXft-2.3.2_1.txz failed dependency check.
gobject-introspection-1.46.0.txz failed dependency check.
libEGL-13.0.4_1.txz failed dependency check.
libglesv2-13.0.4_1.txz failed dependency check.
libXcursor-1.1.14_3.txz failed dependency check.
libXp-1.0.3,1.txz failed dependency check.
dri-13.0.4,2.txz failed dependency check.
libepoxy-1.3.1.txz failed dependency check.
xorg-fonts-truetype-7.7_1.txz failed dependency check.
libXaw-1.0.13,2.txz failed dependency check.
libXcomposite-0.4.4_3,1.txz failed dependency check.
xorg-server-1.18.4,1.txz failed dependency check.
sqlite3-3.16.2.txz failed dependency check.
mpc-1.0.3.txz failed dependency check.
harfbuzz-1.4.1.txz failed dependency check.
libXfontcache-1.0.5_3.txz failed dependency check.
atk-2.18.0.txz failed dependency check.
gdk-pixbuf2-2.32.3_1.txz failed dependency check.
gcc6-aux-20170202_1.txz failed dependency check.
shared-mime-info-1.5.txz failed dependency check.
libXScrnSaver-1.2.2_3.txz failed dependency check.
trapproto-3.4.3.txz failed dependency check.
font-cursor-misc-1.0.3_3.txz failed dependency check.
font-misc-misc-1.1.2_3.txz failed dependency check.
pango-1.38.0_1.txz failed dependency check.
desktop-file-utils-0.22_4.txz failed dependency check.
gtk-update-icon-cache-2.24.29.txz failed dependency check.
libXTrap-1.0.1_3.txz failed dependency check.
libXevie-1.0.3_3.txz failed dependency check.
libXres-1.0.7_3.txz failed dependency check.
libXtst-1.2.3.txz failed dependency check.
libXxf86dga-1.1.4_3.txz failed dependency check.
libXxf86misc-1.0.3_3.txz failed dependency check.
libdmx-1.1.3_3.txz failed dependency check.
liboldX-1.0.1_3.txz failed dependency check.
libxkbui-1.0.2_4.txz failed dependency check.
xauth-1.0.10.txz failed dependency check.
xinit-1.3.4,1.txz failed dependency check.
xinput-1.6.2.txz failed dependency check.
xkbutils-1.0.4.txz failed dependency check.
xmodmap-1.0.9.txz failed dependency check.
xprop-1.2.2.txz failed dependency check.
xrdb-1.1.0.txz failed dependency check.
xset-1.2.3_1.txz failed dependency check.
xsetroot-1.1.1.txz failed dependency check.
xf86-input-keyboard-1.9.0_1.txz failed dependency check.
xf86-input-mouse-1.9.2_1.txz failed dependency check.
xf86-video-ati-7.8.0,1.txz failed dependency check.
xorg-fonts-miscbitmaps-7.7.txz failed dependency check.
py27-sqlite3-2.7.13_7.txz failed dependency check.
autoconf213-2.13.000227_6.txz failed dependency check.
imlib2-1.4.9,2.txz failed dependency check.
libv4l-1.6.3_2.txz failed dependency check.
nss-3.28.1_1.txz failed dependency check.
hunspell-1.6.0.txz failed dependency check.
xmessage-1.0.4.txz failed dependency check.
xorg-apps-7.7_2.txz failed dependency check.
xorg-libraries-7.7_2.txz failed dependency check.
xorg-drivers-7.7_5.txz failed dependency check.
xorg-fonts-7.7_1.txz failed dependency check.
feh-2.18.txz failed dependency check.
mplayer-1.3.0.20161228_3.txz failed dependency check.
rtorrent-0.9.6_1.txz failed dependency check.
mdocml-1.13.4.txz failed dependency check.
firefox-51.0.1_1,1.txz failed dependency check.
rxvt-unicode-9.22.txz failed dependency check.
xorg-7.7_2.txz failed dependency check.
paratype-2.005.txz failed dependency check.
terminus-font-4.40.txz failed dependency check.
fluxbox-1.3.7_1.txz failed dependency check.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
  R => devel/gmake
  R => misc/help2man
  R => print/texinfo
  R => devel/m4
  R => devel/readline
  R => lang/python27
  R => devel/autoconf
  R => devel/automake
  R => devel/libtool
  R => security/libgpg-error
  R => security/libgcrypt
  R => devel/check
  R => textproc/libxslt
  R => x11/xcb-proto
  R => x11/libxcb
  R => x11/libX11
  R => x11/libXext
  R => devel/bison
  R => lang/python2
  R => x11/libXfixes
  R => devel/py-setuptools27
  R => devel/scons
  R => devel/cmake
  R => x11-fonts/fontconfig
  R => x11-toolkits/libXt
  R => x11/libXdamage
  R => x11/libXrender
  R => devel/ninja
  R => x11/libXv
  R => x11/libXvMC
  R => graphics/libdrm
  R => graphics/libglapi
  R => x11/libXi
  R => devel/glib20
  R => x11/libXxf86vm
  R => x11/libxkbfile
  R => x11-toolkits/libXmu
  U => math/gmp (5.1.3_3 => 6.1.2)
  R => math/mpfr
  R => x11/xcb-util
  R => graphics/libGL
  R => x11/xcb-util-renderutil
  R => devel/binutils
  R => devel/nasm
  R => graphics/cairo
  R => graphics/gbm
  R => x11/libXinerama
  R => x11/libXpm
  R => x11/libXrandr
  R => x11/xkbcomp
  R => x11-fonts/libXft
  R => devel/gobject-introspection
  R => graphics/jpeg-turbo
  R => graphics/libEGL
  R => graphics/libglesv2
  R => x11/libXcursor
  R => x11/libXp
  R => x11-fonts/dejavu
  R => graphics/dri
  R => graphics/libepoxy
  R => x11-fonts/xorg-fonts-truetype
  R => x11-toolkits/libXaw
  R => devel/yasm
  R => x11/libXcomposite
  R => x11-servers/xorg-server
  R => databases/sqlite3
  R => math/mpc
  R => print/harfbuzz
  R => shells/bash
  R => x11-fonts/libXfontcache
  R => accessibility/atk
  R => graphics/gdk-pixbuf2
  R => lang/gcc6-aux
  R => misc/shared-mime-info
  R => x11/libXScrnSaver
  R => x11/trapproto
  R => x11-fonts/font-cursor-misc
  R => x11-fonts/font-misc-misc
  R => x11-toolkits/pango
  R => archivers/zip
  R => audio/libogg
  R => devel/desktop-file-utils
  R => devel/ncurses
  R => graphics/giflib
  R => graphics/gtk-update-icon-cache
  U => multimedia/ffmpeg (3.2.3,1 => 3.2.4,1)
  R => x11/libXTrap
  R => x11/libXevie
  R => x11/libXres
  R => x11/libXtst
  R => x11/libXxf86dga
  R => x11/libXxf86misc
  R => x11/libdmx
  R => x11/liboldX
  R => x11/libxkbui
  R => x11/xauth
  R => x11/xinit
  R => x11/xinput
  R => x11/xkbutils
  R => x11/xmodmap
  R => x11/xprop
  R => x11/xrdb
  R => x11/xset
  R => x11/xsetroot
  R => x11-drivers/xf86-input-keyboard
  R => x11-drivers/xf86-input-mouse
  R => x11-drivers/xf86-video-ati
  R => x11-fonts/xorg-fonts-miscbitmaps
  R => audio/libvorbis
  R => databases/py-sqlite3
  R => devel/adacurses
  R => devel/autoconf213
  R => graphics/imlib2
  R => misc/ini_file_manager
  R => multimedia/libv4l
  R => net-p2p/libtorrent
  R => security/nss
  R => textproc/hunspell
  R => x11/xmessage
  R => x11/xorg-apps
  R => x11/xorg-libraries
  R => x11-drivers/xorg-drivers
  R => x11-fonts/xorg-fonts
  R => x11-toolkits/gtk20
  R => graphics/feh
  R => multimedia/mplayer
  R => net/chrony
  R => net-p2p/rtorrent
  R => shells/mksh
  R => sysutils/cdrtools
  R => textproc/mdocml
  R => www/firefox
  R => x11/rxvt-unicode
  R => x11/xorg
  R => x11-fonts/paratype
  R => x11-fonts/terminus-font
  R => x11-wm/fluxbox
Total packages that would be built: 137
The complete build list can also be found at:
/tmp/synth_status_results.txt

The only ffmpeg port requires rebilding the whole system!

What I like in synth:
- building of ports in parallel
- using tmpfs, speeding up building process a bit
- reusing build dependencies, so I don't have to complile them all the time after pkg autoremove
- using of nullfs for sandboxing the building processes

What I don't like:
- see copypaste above
- unlike pkg utility, synth has huge build dependency (consequence of ADA language), hence bootstrap process is time and cpu consuming + you have to compile and keep gcc-aux (while FreeBSD defaults to clang)
- ncurses dependency is also controversial (let me again give pkg as an example of close to ideal package manager, what a pity it doesn't handle ports, ...dreaming) (ideal will be pkgmk from CRUX Linux )
- it doesn't use system /etc/make.conf file so I have to have 2 copies one for make another for synth
- inability to build everywhere, I have to change directory from time to time and that irritates to some degree
- small ports changes (regarding to installed ports) renders sometimes huge number of rebuilds, again time and cpu consuming process

 

[marino]

the "huge build dependency" is a misdirection because building it from scratch is neither required nor recommended.
"pkg ins synth" installs a prebuilt version that works perfectly fine.
There's no "ncurses" dependency because that's statically compiled. You're only seeing that because you
CAVEAT: once synth is installed and then an update is available, then you'll get the full rebuild.
The big one is gcc6-aux which builds in less than 11 minutes on my (now old) core i5.
If you put it against all 27,000 ports, 11-minute build time is nothing remarkable.

"- it doesn't use system /etc/make.conf file so I have to have 2 copies one for make another for synth"
1) blind copying leads to errors. You should put some thought into the optional profile make.conf
2) You don't realize it, but each profile needs it's own make.conf. Profiles can be widely divergent and using another profiles make.conf or the COMPLETELY UNRELATED host system make.conf is wrong
You realize that repositories are meant for more than 1 server right? The host server is just building the packages. You have to control the package building environment exactly depending on the clients. You might have 1 client (the host system) but you don't design a many-to-1 relationship like that.
In other words, using the host system /etc/make.conf is WRONG. Counting it as a downside indicates lack of understanding of how exactly synth is intended to be used.

"- inability to build everywhere, I have to change directory from time to time and that irritates to some degree"
blame FreeBSD itself.

"- small ports changes (regarding to installed ports) renders sometimes huge number of rebuilds, again time and cpu consuming process"
blame
A) rebuilding too frequently
B) the ports process that allows daily trickling of perl and python updates. Many p5 ports cause 14,000 ports to rebuild and they are updated all the time without warning. If I were in charge, perl would be updated enmasse every 30 days. Other ports with such heavy consequences would also be limited. Nobody accounts for this.

 

[ASX]

Phew!

What I don't like:
...
- it doesn't use system /etc/make.conf file so I have to have 2 copies one for make another for synth

No, you should not use make anymore ... that would defeat the benefits of using synth.

- inability to build everywhere, I have to change directory from time to time and that irritates to some degree

it is a nuisance ... and coming from the habit of being in /usr/ports ... after a few days you will learn to remain in your root home directory ...

- small ports changes (regarding to installed ports) renders sometimes huge number of rebuilds, again time and cpu consuming process

Yep, this is a feature coming with some overhead ... I for one prefer to pay the overhead vs. having issues because of missed rebuilds.

 

[kpect]

The big one is gcc6-aux which builds in less than 11 minutes on my (now old) core i5.
If you put it against all 27,000 ports, 11-minute build time is nothing remarkable.

Nontheless it will be rebuilded anytime synth will consider it worthwhile, as in my case above (for the only port that matters)
And not everyone has Corei5 even now. I have one though.

2) You don't realize it, but each profile needs it's own make.conf. Profiles can be widely divergent and using another profiles make.conf or the COMPLETELY UNRELATED host system make.conf is wrong
You realize that repositories are meant for more than 1 server right? The host server is just building the packages. You have to control the package building environment exactly depending on the clients. You might have 1 client (the host system) but you don't design a many-to-1 relationship like that.
In other words, using the host system /etc/make.conf is WRONG. Counting it as a downside indicates lack of understanding of how exactly synth is intended to be used.

I pretty well understand that repos can be used for more that 1 server and I have understanding of why there is a separate make file for each profile. But I judge from a standpoint of a desktop user, not a server maintainer (sysadmin), though I maintain a bunch of SLES, RHEL, HP-UX, debian-based servers as a daily job. From my point of view a default option should stay system /etc/make.conf file but when you want to enable separate profiles you should have possibility to switch them on for example in ini file.

"- inability to build everywhere, I have to change directory from time to time and that irritates to some degree"
blame FreeBSD itself.

Wasn't it possible to change the directory in a subprocess (pushd) and after building return back (popd) ?

"- small ports changes (regarding to installed ports) renders sometimes huge number of rebuilds, again time and cpu consuming process"
blame
A) rebuilding too frequently
B) the ports process that allows daily trickling of perl and python updates. Many p5 ports cause 14,000 ports to rebuild and they are updated all the time without warning. If I were in charge, perl would be updated enmasse every 30 days. Other ports with such heavy consequences would also be limited. Nobody accounts for this.

A) maybe you're right, I have to delay updating at least once per month no more, as I'll have to rebuild the whole system almost always.
B) I agree with you.

No, you should not use make anymore ... that would defeat the benefits of using synth.

He he, why not, I'm thinking of writing some simple scripts which would utilize make and installing everything a-la portmaster way. If some package will be broken because of old dependency (I've bumped into that a couple of times) I'll rebuild everything from scratch, like with portmaster -af.

 

[ASX]

He he, why not

a) synth build in a clean environment, make does not and as a consequence result may vary ...
b) some breakage may be very hard to detect (changed or obsoleted options) ...
c) I can't see why you can't make use a a pkg-list and use synth install <list> or something similar to obtain the same result.

 

[kpect]

I can't see why you can't make use a a pkg-list and use synth install <list> or something similar to obtain the same result.

I'll use synth for several months It's too early to make a final decision, possibly updating once per several weeks will do fine for me.

 

[marino]

Nontheless it will be rebuilded anytime synth will consider it worthwhile, as in my case above (for the only port that matters)
And not everyone has Corei5 even now. I have one though.

Usually, this only happens on major GCC updates (e.g. 6.1 => 6.2 => 6.3).
Recently I added support for aarch64-*-freebsd* and that required using a more recent snapshot than the 6.3 release.
This is a rare event.
Now that I have been kicked out of FreeBSD, chances are gcc6-aux will never be updated again.
So this particular issue is not really valid.

From my point of view a default option should stay system /etc/make.conf file but when you want to enable separate profiles you should have possibility to switch them on for example in ini file.

The problem is that this is just wrong.
/etc/make.conf is primarily for the base building. Re-using it for ports was ultimately a terrible technical decision. There can be build flags/settings perfectly reasonable for the base and completely wrong from ports. That fact that synth and poudriere allows these to be correctly split should only be viewed as the right thing to do. Making a profile-specific make.conf file isn't particularly overdemanding either.

Wasn't it possible to change the directory in a subprocess (pushd) and after building return back (popd) ?

(cd / && synth just-build editors/joe) works from anywhere. I use that trick all the time.

A) maybe you're right, I have to delay updating at least once per month no more, as I'll have to rebuild the whole system almost always.

I think building once per week right after the FreeBSD cluster starts its packages (i don't know the day that is) is reasonable.
I wish they would publish the svn revision each current build is so that people can sync ports to packages.

 

[kpect]

After my last post I've decided to update the system at last, but something went wrong:

$ doas synth upgrade-system                                                                                                                      [ /usr/home/x ]
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
00:00:02 => [02] Builder launched
00:00:02 => [01] Builder launched
00:00:02 => [03] Builder launched
00:00:02 => [01]          Kickoff devel/gmake
00:00:02 => [02]          Kickoff devel/readline
00:00:02 => [03]          Kickoff devel/gettext-runtime
00:00:03 => [03] 00:00:01 Failure devel/gettext-runtime
00:00:03 => [03]          Kickoff archivers/libarchive
00:00:03 => [01] 00:00:01 Failure devel/gmake
00:00:03 => [02] 00:00:01 Failure devel/readline
00:00:03 => [01]          Kickoff misc/pciids
00:00:03 => [02]          Kickoff devel/yasm
00:00:04 => [03] 00:00:01 Failure archivers/libarchive
00:00:04 => [03]          Kickoff graphics/giflib
00:00:04 => [01] 00:00:01 Failure misc/pciids
00:00:04 => [02] 00:00:01 Failure devel/yasm
00:00:04 => [01]          Kickoff archivers/zip
00:00:04 => [02]          Kickoff audio/libogg
00:00:05 => [03] 00:00:01 Failure graphics/giflib
00:00:05 => [03]          Kickoff devel/ncurses
00:00:05 => [01] 00:00:01 Failure archivers/zip
00:00:05 => [02] 00:00:01 Failure audio/libogg
00:00:05 => [01]          Kickoff devel/libslang2
00:00:05 => [02]          Kickoff net-p2p/libtorrent
00:00:06 => [03] 00:00:01 Failure devel/ncurses
00:00:06 => [03]          Kickoff shells/mksh
00:00:07 => [01] 00:00:01 Failure devel/libslang2
00:00:07 => [02] 00:00:01 Failure net-p2p/libtorrent
00:00:07 => [01]          Shutting down
00:00:07 => [02]          Shutting down
00:00:07 => [03] 00:00:01 Failure shells/mksh
00:00:07 => [03]          Shutting down


The task is complete.  Final tally:
Initial queue size: 150
    packages built: 0
           ignored: 0
           skipped: 138
            failed: 12

Duration: 00:00:06
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 217 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%    260 B   0.3kB/s    00:01   
Fetching packagesite.txz: 100%   23 KiB  23.6kB/s    00:01   
Processing entries: 100%
Synth repository update completed. 103 packages processed.
Checking for upgrades (26 candidates): 100%
Processing candidates (26 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

Excerpt from the logs:
--------------------------------------------------------------------------------
--  Phase: pkg-depends
--------------------------------------------------------------------------------
===>   gmake-4.2.1_1 depends on file: /usr/local/sbin/pkg - not found
===>   gmake-4.2.1_1 depends on package: /xports/ports-mgmt/pkg/pkg-1.9.4_1.txz - not found
===>   USE_PACKAGE_DEPENDS_ONLY set - not building missing dependency from source
*** Error code 1

Stop.
make: stopped in /xports/devel/gmake

--------------------------------------------------------------------------------
--  Phase: pkg-depends
--------------------------------------------------------------------------------
===>   yasm-1.3.0 depends on file: /usr/local/sbin/pkg - not found
===>   yasm-1.3.0 depends on package: /xports/ports-mgmt/pkg/pkg-1.9.4_1.txz - not found
===>   USE_PACKAGE_DEPENDS_ONLY set - not building missing dependency from source
*** Error code 1

Stop.
make: stopped in /xports/devel/yasm


$ ll /usr/local/sbin/pkg                                                                                                                         [ /usr/home/x ]
-rwxr-xr-x  1 root  wheel  182824 28 янв.  15:08 /usr/local/sbin/pkg

Dunno why synth couldn't find /usr/local/sbin/pkg binary, though it does exist. I've tried my best to find the reason but had to reinstall the portmaster back and it did his job well as always. I've updated the system. Unfortunately I had to remove the synth for now as it just didn't cope well with updating my OS.