Using synth as a builder for many machines

Well, for now, just one machine running x11/gnome3 and a few other 'apps'. I still don't really understand how it all comes together so this is what I did.

On my builder machine:
Code:
Synth configuration profile: GNOME3
===============================================================================
   [A] Ports directory            /usr/ports
   (B) Packages directory         /var/synth/gnome3_packages
   [C] Distfiles directory        /usr/ports/distfiles
   [D] Port options directory     /var/db/ports_synth_gnome3
   [E] Build logs directory       /var/log/synth_gnome3
   [F] Build base directory       /usr/obj/synth_gnome3
   [G] System root directory      /
   [H] Compiler cache directory   disabled
   (I) Num. concurrent builders   4
   [J] Max. jobs per builder      2
   [K] Use tmpfs for work area    true
   [L] Use tmpfs for /usr/local   true
   [M] Display using ncurses      true
   [N] Fetch prebuilt packages    true

I then used a syncing program to transfer files between the two machines. So:
Builder machine ----------------------------------- Old slow poke machine
/var/db/ports_synth_gnome3 = /var/db/ports
/var/synth/gnome3_packages = var/synth/live_packages

Only reason I used var/synth/live_packages because its existing on the old machine and I haven't removed synth from it yet.

On the builder machine I basically fed it a "just-build" lis which are the main ones, and a few leaf ports:
Code:
x11/xorg
x11/gdm
x11/gnome3
ftp/filezilla
games/minecraft-client
games/warsow
games/wesnoth
games/xonotic
www/firefox
x11/nvidia-driver-340
x11/nvidia-settings

Synth fetched and built a very long list of ports. Then after completion I did pkg upgrade on the old machine. I got two unexpected results: (1) pkg used some of the packages made by synth, and it also fetched a fair few packages from FreeBSD. Which I thought hmmm okay. After it was all done I decided to run synth status on the old machine just as a check to see. To my surprise it said it would download 145 packages and build 14 packages.

Before launching the build job on the builder machine I did a portsnap fetch update on both machines at the exact same time, so both machines have the same ports tree. And the ports options directory are in sync.

So my question is why? Why would synth on the old machine still want to do a whole bunch of work right after the builder machine finished up? Where is my mistake? What other key source of info is missing so that the true state of the old machine is pulled into synth on the builder machine?
 
Chances are you have CONSERVATIVE_UPGRADE set to yes on pkg. (see pkg -vv | grep CONS to verify)
To exclude FreeBSD, you can always call pkg(8) directly: pkg update -r Synth
Or even delete all package and reinstall only from Synth repo to make the setting of CONSERVATIVE_UPGRADE moot.
 
Last edited by a moderator:
I really didn't think you would reply. Thank you. You're a good man. I'll give that a try and fart around some more. I see now too I missed some 'root' ports.
 
A synth status as a sanity check still spews a lot of fetching and rebuilding, immediately after a pkg update -r Synth, and a pkg upgrade -r Synth. Tired, will pick at this again later. Long day; up at 5, now midnight.

#synth status
Code:
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of devel/gmake-lite failed (port deleted), it will not be considered.
Scan of textproc/clucene-qt5 failed (port deleted), it will not be considered.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
Scanning existing packages.   
These are the packages that would be fetched:
  => linux_base-c6-6.8_6.txz (emulators/linux_base-c6)
  => qt5-qmake-5.6.2.txz (devel/qmake5)
  => qt5-buildtools-5.6.2.txz (devel/qt5-buildtools)
  => qt5-core-5.6.2_1.txz (devel/qt5-core)
  => qt5-network-5.6.2.txz (net/qt5-network)
  => qt5-dbus-5.6.2.txz (devel/dbus-qt5)
  => xcb-util-image-0.4.0_1.txz (x11/xcb-util-image)
  => xcb-util-keysyms-0.4.0_1.txz (x11/xcb-util-keysyms)
  => xcb-util-wm-0.4.1_3.txz (x11/xcb-util-wm)
  => qt5-gui-5.6.2_1.txz (x11-toolkits/qt5-gui)
  => linux-c6-expat-2.0.1_2.txz (textproc/linux-c6-expat)
  => linux-c6-fontconfig-2.8.0_1.txz (x11-fonts/linux-c6-fontconfig)
  => qt5-widgets-5.6.2.txz (x11-toolkits/qt5-widgets)
  => linux-c6-xorg-libs-7.4_6.txz (x11/linux-c6-xorg-libs)
  => qt5-sql-5.6.2.txz (databases/qt5-sql)
  => qt5-testlib-5.6.2.txz (devel/qt5-testlib)
  => linux-c6-jpeg-1.2.1_1.txz (graphics/linux-c6-jpeg)
  => linux-c6-png-1.2.49_3.txz (graphics/linux-c6-png)
  => qt5-qml-5.6.2.txz (lang/qt5-qml)
  => qt5-xmlpatterns-5.6.2.txz (textproc/qt5-xmlpatterns)
  => snappy-1.1.3.txz (archivers/snappy)
  => linux-c6-pixman-0.32.8.txz (x11/linux-c6-pixman)
  => qt5-quick-5.6.2.txz (x11-toolkits/qt5-quick)
  => qt5-opengl-5.6.2.txz (graphics/qt5-opengl)
  => linux-c6-libthai-0.1.12.txz (devel/linux-c6-libthai)
  => linux-c6-nspr-4.11.0.txz (devel/linux-c6-nspr)
  => tradcpp-0.5.2.txz (devel/tradcpp)
  => qt5-printsupport-5.6.2.txz (print/qt5-printsupport)
  => linux-c6-cairo-1.8.8_6.txz (graphics/linux-c6-cairo)
  => linux-c6-jasper-1.900.1_1.txz (graphics/linux-c6-jasper)
  => linux-c6-tiff-3.9.4_2.txz (graphics/linux-c6-tiff)
  => leveldb-1.19.txz (databases/leveldb)
  => linux-c6-sqlite-3.6.20_3.txz (databases/linux-c6-sqlite3)
  => qt5-sqldrivers-sqlite3-5.6.2.txz (databases/qt5-sqldrivers-sqlite3)
  => gccmakedep-1.0.3.txz (devel/gccmakedep)
  => xorg-cf-files-1.0.6.txz (x11/xorg-cf-files)
  => linux-c6-atk-1.30.0.txz (accessibility/linux-c6-atk)
  => linux-c6-pango-1.28.1_6.txz (x11-toolkits/linux-c6-pango)
  => linux-c6-cyrus-sasl2-2.1.23_3.txz (security/linux-c6-cyrus-sasl2)
  => linux-c6-nss-3.21.3.txz (security/linux-c6-nss)
  => linux-c6-openssl-1.0.1e_11.txz (security/linux-c6-openssl)
  => linux-c6-gdk-pixbuf2-2.24.1_3.txz (graphics/linux-c6-gdk-pixbuf2)
  => lua51-5.1.5_9.txz (lang/lua51)
  => tcl86-8.6.6_1.txz (lang/tcl86)
  => qt5-webkit-5.6.2_1.txz (www/webkit-qt5)
  => imake-1.0.7,1.txz (devel/imake)
  => linux-c6-libelf-0.164.txz (devel/linux-c6-libelf)
  => linux-c6-libpciaccess-0.13.4_1.txz (devel/linux-c6-libpciaccess)
  => swig-2.0.12_1.txz (devel/swig20)
  => p5-Unicode-String-2.10.txz (converters/p5-Unicode-String)
  => linux-c6-gtk2-2.24.23_5.txz (x11-toolkits/linux-c6-gtk2)
  => pangox-compat-0.0.2_1.txz (x11-toolkits/pangox-compat)
  => qt5-uiplugin-5.6.2.txz (x11-toolkits/qt5-uiplugin)
  => tk86-8.6.6.txz (x11-toolkits/tk86)
  => linux-c6-libssh2-1.4.2_5.txz (security/linux-c6-libssh2)
  => linux-c6-openldap-2.4.40_2.txz (net/linux-c6-openldap)
  => linux-c6-libogg-1.1.4_2.txz (audio/linux-c6-libogg)
  => qt5-xml-5.6.2.txz (textproc/qt5-xml)
  => apr-1.5.2.1.5.4_2.txz (devel/apr1)
  => gamin-0.1.10_8.txz (devel/gamin)
  => linux-c6-sdl-1.2.14_3.txz (devel/linux-c6-sdl12)
  => p8-platform-2.0.1.txz (devel/p8-platform)
  => qt5-script-5.6.2.txz (devel/qt5-script)
  => swig30-3.0.10_1.txz (devel/swig30)
  => jakarta-commons-codec-1.10.txz (java/jakarta-commons-codec)
  => jakarta-commons-logging-1.2.txz (java/jakarta-commons-logging)
  => gnome-mime-data-2.18.0_5.txz (misc/gnome-mime-data)
  => c-ares-1.12.0.txz (dns/c-ares)
  => libirman-0.4.6.txz (comms/libirman)
  => p5-Unicode-Map8-0.13_1.txz (converters/p5-Unicode-Map8)
  => recode-3.6_12.txz (converters/recode)
  => Xaw3d-1.5E_8.txz (x11-toolkits/Xaw3d)
  => py27-tkinter-2.7.12_6.txz (x11-toolkits/py-tkinter)
  => linux-c6-openssl-compat-0.9.8e_4.txz (security/linux-c6-openssl-compat)
  => linux-c6-libvorbis-1.2.3_2.txz (audio/linux-c6-libvorbis)
  => linux-c6-curl-7.19.7_8.txz (ftp/linux-c6-curl)
  => linux-c6-dri-11.0.7_3.txz (graphics/linux-c6-dri)
  => python34-3.4.5.txz (lang/python34)
  => redeclipse-data-1.5.3.txz (games/redeclipse-data)
  => gsed-4.2.2.txz (textproc/gsed)
  => rasqal-0.9.33.txz (textproc/rasqal)
  => 9base-b20100604.txz (devel/9base)
  => ChipmunkPhysics-7.0.1.txz (devel/ChipmunkPhysics)
  => electricfence-2.2.2_2.txz (devel/ElectricFence)
  => dmake-4.12.20150309.txz (devel/dmake)
  => epm-4.3.txz (devel/epm)
  => gettext-0.19.8.1.txz (devel/gettext)
  => gperf-3.0.3_1.txz (devel/gperf)
  => libplist-1.12.txz (devel/libplist)
  => libsigsegv-2.10_1.txz (devel/libsigsegv)
  => llvm35-3.5.2_2.txz (devel/llvm35)
  => p5-IO-HTML-1.001_1.txz (devel/p5-IO-HTML)
  => patch-2.7.5.txz (devel/patch)
  => py27-logilab-common-1.2.2.txz (devel/py-logilab-common)
  => py34-setuptools34-28.1.0.txz (devel/py-setuptools34)
  => qt5-concurrent-5.6.2.txz (devel/qt5-concurrent)
  => upnp-1.6.20,1.txz (devel/upnp)
  => bsdadminscripts-6.1.1_8.txz (sysutils/bsdadminscripts)
  => jakarta-commons-httpclient-3.1_1.txz (java/jakarta-commons-httpclient)
  => xf86-video-fbdev-0.4.4_7.txz (x11-drivers/xf86-video-fbdev)
  => cmatrix-1.2a_1.txz (misc/cmatrix)
  => qt5-serialport-5.6.2.txz (comms/qt5-serialport)
  => enca-1.19.txz (converters/enca)
  => CoinMP-1.8.3.txz (math/coinmp)
  => dialog4ports-0.1.6.txz (ports-mgmt/dialog4ports)
  => croscorefonts-1.31.0.txz (x11-fonts/croscorefonts-fonts-ttf)
  => crosextrafonts-caladea-20130214_1.txz (x11-fonts/crosextrafonts-caladea-ttf)
  => crosextrafonts-carlito-20130920_2.txz (x11-fonts/crosextrafonts-carlito-ttf)
  => ffmpeg0-0.7.17_6,1.txz (multimedia/ffmpeg0)
  => libass-0.13.4.txz (multimedia/libass)
  => libcec-3.1.0.txz (multimedia/libcec)
  => libmpeg2-0.5.1_6.txz (multimedia/libmpeg2)
  => librtmp-2.4.20151223.txz (multimedia/librtmp)
  => qt5-multimedia-5.6.2.txz (multimedia/qt5-multimedia)
  => mDNSResponder-765.1.2.txz (net/mDNSResponder)
  => i386-wine-1.8.5,1.txz (emulators/i386-wine)
  => libsamplerate-0.1.9.txz (audio/libsamplerate)
  => libshairport-1.2.0.20310_2.txz (audio/libshairport)
  => lcms-1.19_6,1.txz (graphics/lcms)
  => py27-pillow-3.1.1_1.txz (graphics/py-pillow)
  => silgraphite-2.3.1_4.txz (graphics/silgraphite)
  => bsh-2.0.b6.txz (lang/bsh)
  => 3dc-0.8.1_3.txz (games/3dc)
  => briquolo-0.5.7_13.txz (games/briquolo)
  => bzflag-2.4.6.txz (games/bzflag)
  => linux-defcon-1.42_2.txz (games/linux-defcon)
  => asciidoc-8.6.9_5.txz (textproc/asciidoc)
  => hyphen-2.8.8.txz (textproc/hyphen)
  => libtextcat-2.2_6.txz (textproc/libtextcat)
  => mythes-1.2.4.txz (textproc/mythes)
  => redland-1.0.17_4.txz (textproc/redland)
  => tinyxml-2.6.2_1.txz (textproc/tinyxml)
  => libmicrohttpd-0.9.37_2.txz (www/libmicrohttpd)
  => serf-1.3.9_1.txz (www/serf)
Total packages that would be fetched: 134
The complete build list can also be found at:
/tmp/synth_prefetch_list.txt
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => devel/qt5-help
  N => devel/qt5-assistant
  N => devel/libftdi
  N => devel/qt5-designer
  N => x11-toolkits/wxgtk28-common
  N => www/libxul
  N => devel/gnome-vfs
  N => java/icedtea-web
  N => java/openjdk7-jre
  N => comms/lirc
  N => x11-toolkits/qt5-declarative
  N => x11-toolkits/wxgtk28
  N => graphics/libopenraw
  N => games/redeclipse
  N => www/linux-c6-flashplugin11
  N => www/nspluginwrapper
Total packages that would be built: 16
The complete build list can also be found at:
/tmp/synth_status_results.txt
[/U]
 
manually remove these two with pkg remove
Code:
Scan of devel/gmake-lite failed (port deleted), it will not be considered.
Scan of textproc/clucene-qt5 failed (port deleted), it will not be considered.

The downloaded files are probably build dependencies that won't get installed or even used.
Yes I know it's a waste to download something that's not used, but it's a side effect of the "fetch prebuilt packages".

After building those 16 with synth prepare-system followed by synth upgrade-system upon success, I imagine synth status will then be happy as long as you don't update the ports tree.
 
Sweet, it works!

I left prefetching enabled on the builder machine, but I turned it off for the target recipient machine. And I configured pkg
Code:
CONSERVATIVE_UPGRADE = false;
on the target machine. (And I installed and got devel/ccache on the builder for performance gains.)

After building those 16 with synth prepare-system followed by synth upgrade-system upon success, I imagine synth status will then be happy as long as you don't update the ports tree.

I don't want to use synth to do the upgrading on the target machine, I want to use pkg. (But I might keep using synth we'll see). The goal is to remove synth from the old slow poke machines. And I don't think you want me to synth upgrade-system on the builder machine because that will actually install the ports there right? Anyway I got through it. It took a few multiple passes, but it worked. I did the 16 as you suggested, and then another pass suggested another 67; and I did these using a file with the ports listed; so like this #synth just-build synth_temp_list.txt.

After completed I did one more pass around and got here:

# synth rebuild-repository
Code:
Stand by, prescanning existing packages.
Stand by, recursively scanning 1026 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt

# synth status
Code:
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt

# pkg upgrade (note nothing is actually upgraded)
Code:
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (252 candidates): 100%
Processing candidates (252 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

# synth status
Code:
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt

Sweet! I think the root cause of my issue was the initial seeding of synth on the builder machine with all the ports. Is there a command that will list all the installed ports on a machine in the format category/port-name (without the -xx.x.x,x version info) that I can easily feed back into synth? I can't find a portmaster option to do it. And note, you need to ensure the syncing program has finished and synchronization has been achieved.

When I buy my next new machine it too will likely be GNOME3 desktop. I'll simply add it to the syncing program. I use
net-p2p/btsync
which works well.

Thanks again.
 
Is there a command that will list all the installed ports on a machine in the format category/port-name (without the -xx.x.x,x version info) that I can easily feed back into synth?.

pkg query -e '%a==0' '%o' > ~/my.build.list will create it in the format you want.
I should get that recipe added to pkg(8) default aliases as "prime-origins"
 
So this is all working great it seems. But one last question I think. I use the same process to build package that I want to install. Trouble is pkg won't install from Synth respository, pkg wants to use the FreeBSD respository. I tried some variations of the pkg command to no avail. I'm guessing I need to use a flag or to, and maybe manually specify the directory full path. Will try that later. Any thoughts?

# ls -l | grep obs
Code:
-rw-r--r--  1 btsync  wheel     2673944 Nov 30 19:57 obs-studio-0.16.0.txz

# pkg add obs-studio
Code:
pkg: obs-studio: No such file or directory
pkg: Was 'pkg install obs-studio' meant?

# pkg install obs-studio
Code:
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 7 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   obs-studio: 0.16.0 [FreeBSD]
   jackit: 0.124.1_5 [FreeBSD]
   libsysinfo: 0.0.3 [FreeBSD]
   qt5-imageformats: 5.6.2 [FreeBSD]
   libmng: 1.0.10_2 [FreeBSD]
   qt5-x11extras: 5.6.2 [FreeBSD]
   fdk-aac: 0.1.4 [FreeBSD]

Number of packages to be installed: 7


pkg install -R Synth obs-studio
Code:
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 7 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   obs-studio: 0.16.0 [FreeBSD]
   jackit: 0.124.1_5 [FreeBSD]
   libsysinfo: 0.0.3 [FreeBSD]
   qt5-imageformats: 5.6.2 [FreeBSD]
   libmng: 1.0.10_2 [FreeBSD]
   qt5-x11extras: 5.6.2 [FreeBSD]
   fdk-aac: 0.1.4 [FreeBSD]


# pkg -vv

Code:
Repositories:
  Synth: {
    url             : "file:///var/synth/live_packages",
    enabled         : yes,
    priority        : 0
  }
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }

CONSERVATIVE_UPGRADE = false;

Lastly, I tried to install using Synth and it works without issue:
# synth install multimedia/obs-studio
Code:
New packages to be INSTALLED:
   obs-studio: 0.16.0 [Synth]
   jackit: 0.124.1_5 [Synth]
   libsysinfo: 0.0.3 [Synth]
   qt5-imageformats: 5.6.2 [Synth]
   libmng: 1.0.10_2 [Synth]
   qt5-x11extras: 5.6.2 [Synth]
   fdk-aac: 0.1.4 [Synth]

Number of packages to be installed: 7
 
This is incorrect. It should be pkg install -r Synth obs-studio with lowercase 'r'. Check man pkg-install and you'll see why your command didn't work since 'R' is recursive and 'r' is reponame.

pkg install -R Synth obs-studio

Priority number should be higher for Synth than FreeBSD. ie, 100.
Code:
Repositories:
  Synth: {
    url             : "file:///var/synth/live_packages",
    enabled         : yes,
    priority        : 0
  }
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }

CONSERVATIVE_UPGRADE = false;
 
It should be pkg install -r Synth obs-studio
Priority number should be higher for Synth than FreeBSD. ie, 100.

I did both and neither worked. I was wondering about the priority number. But the strange thing is, I uninstalled the package and then tried both again just to view the results again. This time was different, the -r option worked. I musta typoed.
 
Back
Top