Synth 1.30 has been released

Code:
ports-mgmt/synth: Upgrade version 1.22 => 1.30

This is an enhancement release that contains a bug fix.

Description of bug:
  If synth is launched from mountpoint of the ports directory
  (e.g. /usr/ports), it may malfunction with strange messages such
  as "invalid origins" and printing strange symbols to the screen
  along with parts of a directory.

  As a result, Synth now detects this launch location (as it already
  did with /usr/local) and refuses to run until the current directory
  is changed outside of the ports tree.

Enhancement 1:
  It is now possible to remove alternate profiles.  When more than
  one profile exists, a menu option "<" appears that provides the
  user with an opportunity to remove one to all inactive profiles
  from the configuration.  The man page has been updated as well.

Enhancement 2:
  Synth will automatically convert any directory inputs on the
  configure command to the true path.  For example, if somebody types
  in "/usr/xports" for the ports directory, but that's just a
  symbolic link to /vcs/freebsd-ports", the entry will automatically
  convert to its true resolved path (e.g. /vcs/freebsd-ports). This
  enables the cwd detection of the bug fix above to always work.
 
FYI, there was one previously unannounced bug fix before it:

Code:
ports-mgmt/synth: Upgrade version 1.21 => 1.22

Two minor bug fixes:
* A specific check during test mode would emit a failure to stdout when
   testing devel/py-setuptools27.  It turns out that there's a file there
   with a space in the filename.  The filename was an argument for
   /usr/bin//file and it wasn't escaped.  The file in question had
   parentheses too which the shell was trying to process.  The fix was
   to escape the filename in the /usr/bin/file command.
* The builders were mounting the source directory from "/usr/src", not
   $sysroot/usr/src as intended.  This potentially causes breakage when
   the $sysroot reflects a different versions/release than the host
   machine has (e.g. making FreeBSD 10.2 packages on FreeBSD 11-current).
   Now the source directory mount is relative to profile's $sysroot.

So definitely upgrade to 1.30, mainly for the ports directory check which prevents nasty output. (Thanks Crivens)
 
I've just installed ports-mgmt/synth 1.30 on a FreeBSD 9.3 machine and on the first run, this is what I get:

Code:
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 1.30
====================================================================
  Copyright (C) 2015-2016 John R. Marino


Usage: synth [zero-parameter-option]
-or-  synth [list-option] <list of port origins | filename>

zero-parameter-option includes 'help', 'configure', 'version' (this screen),
  'status', 'upgrade-system', 'prepare-system',
  'status-everything', 'everything', 'purge-distfiles',
  'rebuild-repository'
list-option includes  'status', 'build', 'just-build', 'install', 'force'
  'test'

# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
mount: /usr/obj/synth-live/SL09/boot/modules: No such file or directory

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs -o size=100M tmpfs /usr/obj/synth-live/SL09/boot/modules => failed with code 64

I've tried again a few seconds after and got:

Code:
# synth status
Builder mounts detected; attempting to remove them automatically ...
piped_mute_command failure:
 => umount: 18G: statfs: No such file or directory
umount: 18G: unknown file system
umount: 9.1G: statfs: No such file or directory
umount: 9.1G: unknown file system
umount: 8.3G: statfs: No such file or directory
umount: 8.3G: unknown file system
umount: 53%: statfs: No such file or directory
umount: 53%: unknown file system
umount: /usr/obj/synth-live/SL09/construction: not a file system root directory

piped_mute_command failure:
 => umount: 18G: statfs: No such file or directory
umount: 18G: unknown file system
umount: 9.1G: statfs: No such file or directory
umount: 9.1G: unknown file system
umount: 8.3G: statfs: No such file or directory
umount: 8.3G: unknown file system
umount: 53%: statfs: No such file or directory
umount: 53%: unknown file system
umount: /usr/obj/synth-live/SL09/usr/local: not a file system root directory

Dismounting successful!
Old work directories detected; attempting to remove them automatically ...
Directory removal successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
mount: /usr/obj/synth-live/SL09/boot/modules: No such file or directory

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs -o size=100M tmpfs /usr/obj/synth-live/SL09/boot/modules => failed with code 64
 
france75006@, you don't have /boot/modules directory ?

if not, create it, but I thought /boot/modules existed on every system.
 
I didn't have that directory (it's an OVH dedicated server), but now that I've created it everything seems ok for synth: thanks!
 
I'm in the process of building a new server with full-disk encryption and /boot on a USB drive. Does this mean I will need to leave the /boot directory mounted whenever I run synth? I had planned to remove the USB drive after each startup and only mount it when changes to /boot were necessary.
 
No, /boot is optional. But if you have /boot, you also have to have /boot/modules. At least, that's true for 1.30 and earlier.

edit: note that certain options (DTRACE) and certain ports (e.g. kernel modules) require /boot and /boot/modules to build, so not having it will ensure those ports do not build. But they are fairly rare.
 
Last edited by a moderator:
Deploying some new machines, and decided to take synth for a test ride...

So:
- I've set up my pkg to latest instead quarterly
- updated and upgraded
- snapped them ports
- there's nothing on the system yet, installed synth
- configured it to leverage_prebuilt=true
- did "synth status" to dry run it, it said that it'll fetch 121 packages and install 5 new ones
- now it's building 126 packages :/

I've kinda hoped it'll fetch prebuilt packages? It's installing nothing that's not readily available via pkg.
 
- did "synth status" to dry run it, it said that it'll fetch 121 packages and install 5 new ones
- now it's building 126 packages :/
I'll take a look at the code. synth status is supposed to be accurate and apparently it was able to query the remote database.
 
Okay, here's my best guess:
Synth uses the host pkg(8) program to prefetch the packages.
This particular command continues quietly when it failed, but I think it should have said "failed to download at least one package" and stopped then.

so if the above is true, and host pkg program was missing, synth needs to abort immediately. Also there should be some indication if the actual command fails. I'll put this on the to do list regardless.
 
although I'm not sure it's possible to have synth installed w/o pkg .... so that might blow the theory ...
 
so far I can't see how the dry run would work but the actual fetching would fail without stopping.
It has a list of packages it must fetch (from the remote query) and all have to be present before Synth starts building.

I'm not saying you're wrong, only that I have to set up a test case and try to reproduce it myself.
 
Synth Meat@,
During my attempt to reproduce, I got a different failure signature. When I put in so print lines, I got a command which I re-ran on the system:
Code:
ironwolf# /usr/local/sbin/pkg rquery -U %q -r Wolfpond uhttpmock-0.5.0
Repository Synth has a wrong packagesite, need to re-create database
pkg: Repository Synth cannot be opened. 'pkg update' required
dragonfly:4.6:x86:64

In my case, a bad "synth" repo is interfering with the external repo check. You might be experiencing the same thing.
It seems pkg(8) is doing something unexpected here -- if the command limits the repository to "Wolfpond", why are errors about other repositories being reported?
 
Hmm, look at what happens when I rearrange the command:
Code:
ironwolf# /usr/local/sbin/pkg rquery -r Wolfpond -U %q uhttpmock-0.5.0
dragonfly:4.6:x86:64
 
No, pkg(8) definitely exists. That's how I installed synth 1.30 in the first place.

I think the command would have worked if used synth prepare-repository. What I think happened is that synth upgrade-system wrote the /usr/local/etc/pkg/repos/00_synth.conf before executing the build command, and due to the pkg rquery issue above, it caused error messages that weren't seen on the previous synth status output.

I've got this fixed in the repo and I'll push 1.31 shortly so you can re-test in your VPS.
 
Code:
ports-mgmt/synth: Upgrade version 1.30 => 1.31

Pkg(8) bug workaround:
  The pkg rquery man page is incorrect.  It lists the invocation of
  pkg rquery as "-U -r reponame" but if it's invoked in that order, it
  will squawk errors if any of the repositories are misconfigured.  The
  command must be "pkg rquery -r reponame -U" in order to limit pkg(8) to
  a single repository.

  The order sensitivity came into play with "synth upgrade-system" command
  on a brand new system.  The 00_synth.conf pkg(8) configuration file was
  created but there were no repository files created yet, cause pkg(8) to
  emit errors on Synth repo when checking the official FreeBSD package
  repo. This resulted in no packages prefetching even though "synth status"
  indicated some would be fetched.  Rearranging the internal rquery
  commands works around the misleading pkg(8) documentation (or bug?).

FreeBSD-specific system root validity check.
  The $SYSROOT/boot directory doesn't have to be present, but if it is,
  there must also be present $SYSROOT/boot/modules directory. Synth now
  checks for this on FreeBSD and stops with an informative message when
  the modules directory needs to be created by the root user.
 
I've discovered some strange behavior when changing the options in a port.

Let's say I have a port installed with the config settings all at defaults. I then change the config file ( make config) and run synth upgrade-system. Synth proceeds to rebuild the port in the repository, but then the updated package doesn't get installed. I then run synth upgrade-system again. The port gets rebuilt again in the repository, but this time the newly-built package is installed in place of the old one.

I've confirmed this behavior on two separate systems with different ports.

Here's the output of Synth right after changing a port option (security/openssh-portable) and then immediately rerunning Synth a second time:

Code:
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
openssh-portable-7.2.p2,1.txz failed option check.
progress: 90.83%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  70 KiB  71.4kB/s  00:01
Processing entries: 100%
Synth repository update completed. 245 packages processed.
Checking for upgrades (13 candidates): 100%
Processing candidates (13 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 87.07%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking for upgrades (14 candidates): 100%
Processing candidates (14 candidates): 100%
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
  openssh-portable-7.2.p2,1 (options changed)

The operation will free 19 KiB.
[1/1] Reinstalling openssh-portable-7.2.p2,1...
You may need to manually remove /usr/local/etc/ssh/sshd_config if it is no longer needed.
[1/1] Extracting openssh-portable-7.2.p2,1: 100%
Message from openssh-portable-7.2.p2,1:
To enable this port, add openssh_enable="YES" in your rc.conf. To
prevent conflict with openssh in the base system add sshd_enable="NO"
in your rc.conf. Also you can configure openssh at another TCP port (via
sshd_config 'Port' and 'Listen' options or via 'openssh_flags'
variable in rc.conf) and run it in same time with base sshd.

'PermitRootLogin no' is the default for the OpenSSH port.
This now matches the PermitRootLogin configuration of OpenSSH in
the base system.  Please be aware of this when upgrading your
OpenSSH port, and if truly necessary, re-enable remote root login
by readjusting this option in your sshd_config.

Users are encouraged to create single-purpose users with ssh keys, disable
Password authentication by setting 'PasswordAuthentication no' and
'ChallengeResponseAuthentication no', and to define very narrow sudo
privileges instead of using root for automated tasks.
This is with Synth 1.31. Is there anything else I can provide that would help debug this?
 
First, this is pkg(8), not synth. Synth just makes the repository; pkg(8) is responsible for updating the system (Synth just spawns a command to pkg(8)).

Secondly, what is the output of pkg -vv | grep CONS?
Is it CONSERVATIVE_UPGRADE = true?

If so, that might the pkg(8) setting causing it.
 
First, this is pkg(8), not synth. Synth just makes the repository; pkg(8) is responsible for updating the system (Synth just spawns a command to pkg(8)).

Secondly, what is the output of pkg -vv | grep CONS?
Is it CONSERVATIVE_UPGRADE = true?

If so, that might the pkg(8) setting causing it.
Yes, CONSERVATIVE_UPGRADE=true is the result, but that's not the problem.

It turns out Synth does not copy the package into the repository after the first build attempt, leaving the repo without any package for that port. It does copy the package on the second attempt, and only then can pkg find it to install it.

Code:
root@wintermute-pts/2(2:08pm)~/:ll /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
-rw-r--r--  1 root  wheel  632400 Mar 19 08:27 /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
root@wintermute-pts/2(2:08pm)~/:make -C /usr/ports/security/openssh-portable config

root@wintermute-pts/2(2:08pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
openssh-portable-7.2.p2,1.txz failed option check.
progress: 86.38%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(2:10pm)~/:ll /var/synth/live_packages/All/openssh-portable*
ls: No match.
root@wintermute-pts/2(2:12pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 85.84%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(2:13pm)~/:ll /var/synth/live_packages/All/openssh-portable*
-rw-r--r--  1 root  wheel  627176 Mar 19 14:12 /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
root@wintermute-pts/2(2:16pm)~/:
As you can see, at first the package exists in the repo -- in this state, there is no config file. I then run make config and change a few settings. I then run synth prepare-system, and once the port is finished building and Synth completes its work, the openssh-portable package file is missing from the repo. I run synth prepare-system again, Synth rebuilds the port a second time (because the package is now missing from the repo), and the package now appears, ready to be installed by pkg.

Can you reproduce this using the above steps?
 
It turns out Synth does not copy the package into the repository after the first build attempt, leaving the repo without any package for that port. It does copy the package on the second attempt, and only then can pkg find it to install it.

This makes no sense. There's no copying, ever. the packages are created in the proper place and the pkg repo command generates the digests, etc.


As you can see, at first the package exists in the repo -- in this state, there is no config file. I then run make config and change a few settings.

This also makes no sense. no config file is the best state to be in. The lack of a config file is a good thing.

I then run synth prepare-system, and once the port is finished building and Synth completes its work, the openssh-portable file is missing from the repo. I run synth prepare-system again, Synth rebuilds the port a second time (because the package is now missing from the repo), and the package now appears, ready to be installed by pkg.

Did you notice the openssh-portable file got removed during the repositiory creation because it was invalid?
The opposite of what you think is happening. It's in place, but then removed because it fails validity checks.

If openssh-portable gets removed repeatedly, there's a problem with the port (e.g. a LIB_DEPENDS that conditionally satisfied). Synth is doing what it's supposed to do. It doesn't do a full repository validation until right before the repo is created, and packages get deleted in the process (sometimes a lot of packages)
 
This also makes no sense. no config file is the best state to be in. The lack of a config file is a good thing.
Now this makes no sense. If I want to disable a default in a port config, or if I have to enable a different setting in a port, then config files are absolutely needed. If I didn't change the configs, I'd rely solely on pkg and not even bother with Synth.

Even if I'm not describing the process properly, Synth is clearly doing something wrong here. Here's another example, and I'll try to make it as clear as possible.

First, let's start with installing a brand-new port, say dns/ddclient because it has only one config option.

Code:
root@wintermute-pts/2(3:01pm)~/:synth install dns/ddclient



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:10
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  ddclient: 3.8.2_1

The process will require 178 KiB more space.
[1/1] Installing ddclient-3.8.2_1...
[1/1] Extracting ddclient-3.8.2_1: 100%
Message from ddclient-3.8.2_1:
***********************************************************

Copy
  /usr/local/etc/ddclient.conf.sample
to
  /usr/local/etc/ddclient.conf

and edit it to fit your needs.

If you would like to run ddclient as a daemon add the
following line to /etc/rc.conf

  ddclient_enable="YES"

If you would like to force ddclient to update your account
daily regardless of IP changes add the following line to
your /etc/periodic.conf

  daily_ddclient_force_enable="YES"

***********************************************************
root@wintermute-pts/2(3:02pm)~/:
So dns/ddclient is now installed, and the package file exists in the repo.

Right now, no port options are manually set:

Code:
root@wintermute-pts/2(3:03pm)~/:ll /var/db/ports/dns_ddclient*
ls: No match.
root@wintermute-pts/2(3:04pm)~/:

Let's change the port's only option, SSL, by disabling it (it's enabled by default):

Code:
root@wintermute-pts/2(3:04pm)~/:make -C /usr/ports/dns/ddclient config

root@wintermute-pts/2(3:05pm)~/:ll /var/db/ports/dns_ddclient*
total 2
-rw-r--r--  1 root  wheel  166 Mar 19 15:05 options
root@wintermute-pts/2(3:05pm)~/:cat /var/db/ports/dns_ddclient/options
# This file is auto-generated by 'make config'.
# Options for ddclient-3.8.2_1
_OPTIONS_READ=ddclient-3.8.2_1
_FILE_COMPLETE_OPTIONS_LIST=SSL
OPTIONS_FILE_UNSET+=SSL
root@wintermute-pts/2(3:05pm)~/:
Now that we've changed a port option, let's rerun Synth and update the repository:

Code:
root@wintermute-pts/2(3:05pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
ddclient-3.8.2_1.txz failed option check.
progress: 86.72%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:07pm)~/:
So dns/ddclient failed the option check when I disabled its one option, and the package should have been rebuilt with the new options selected (or deselected in this case). But the newly-built package is missing from the repository and cannot be installed by pkg.

Code:
root@wintermute-pts/2(3:07pm)~/:ll /var/synth/live_packages/All/ddclient*
ls: No match.
This, as I understand it, is wrong. Why is the package now missing from the repository after it was successfully built with synth prepare-system? Where did it go? Why do I have to run Synth twice to get the package file back in the repo and ready to be installed?

Code:
root@wintermute-pts/2(3:08pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 87.27%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:12pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40584 Mar 19 15:12 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:12pm)~/:

Can you explain to me why this is normal?
 
Back
Top