Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

Hi, my apologies if this has been asked before; I'm using ports-mgmt/synth to maintain a local repository of all packages on my system and a number of jails.

My problem is, that many packages that are used in the jails keep vanishing from the repository when it is upgraded, leaving me to identify and manually instruct synth to rebuild these.
Is it possible to maintain a list of ports to keep upgraded, even if they do not appear to be installed? Or flag manually built ports to be kept updated?

Also, thanks for the work you've put into developing synth. It has worked really well for me.
 
From your description, the best practice is to maintain a text file with a list of ports you want present (a union between all jails and system).
You would not use "prepare-system" which is just a shortcut for creating a build list from current installation.

You can make the first draft of the list with pkg prime-origins (or pkg 'query -e '%a = 0' '%o'), save result and edit.
Then in the future you build packages by passing synth this file to build.

From your description ("packages keep vanishing"), it sounds like you are building ports one-by-one rather than as a integrated group. If you had a static list, no packages would be missing because synth would identify obsolete and build them all. So I can't tell what exactly your process is from your description, but at first blush it sounds it's not the correct one for you tasks.
 
I agree my current approach was not the best. I guess I expected prepare-system to update the entire repository, not just what pkg reported as installed, but reading the docs again it's clear that is what is happening.

My exact process was to first update the ports tree for system portsnap fetch update and jails ezjail-admin update -P, then run synth prepare-system.
Once that had finished, I went through each jail and ran pkg update followed by pkg upgrade.
Then I would notice that some jails still had packages that needed updating. Specifically those that were not installed on the system.
Querying the repository revealed that these were no longer present and I had to instruct synth to build them again.
Once that was done, I could return to the jails and finish the upgrade.
A couple of months later, rinse and repeat.

I figured I was missing something along the lines of pkg set -A 0 <package> to make synth retain ports not installed on the system, but maintaining a list of prime ports is a manageable solution. Especially given the pkg prime-origins command, which I wasn't familiar with.

Thanks for the help. :)
 
I'm new to ports-mgmt/synth (just installed) and previously have been using ports-mgmt/portmaster for years.

My question is - when I run synth status, it shows :

Code: 
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
  N => print/indexinfo
  N => devel/gettext-runtime
...
...
  N => www/nginx
Total packages that would be built: 152
Why does Synth want to build all 152 ports when they are already installed?

I guess I just don't understand why "/var/synth/live_packages" is even necessary? (I'm assuming that's where the ports will be compiled in to)

Do I just need to run synth prepare-system for now since everything is already installed and running?
 
because synth is a repository builder.
Your proposed repository has no packages. The "prepare-system" command is basically telling synth to build (if needed) packages given a list generated by what you have installed currently. So that list is 152 packages long. You have zero packages. 152 - 0 = 152.

Moreover, just building packages doesn't change your host system. At some point you'd have to tell pkg(8) to use the new repository to update the system. Basically your confusion seems to be caused by a fundamental misunderstanding of what Synth does and how it interacts with pkg(8). Don't worry, it's a common misunderstanding.
 
marino said:
At some point you'd have to tell pkg(8) to use the new repository to update the system.
Click to expand...

Ok, that clears up the confusion. I've only used PKG to delete ports and to check for port dependencies.

I've never used it to update the system, so this is something I'll need to figure out. I stayed away from it because it required messing with the config file (and point to a repository + other things I didn't understand).
 
I'm certain you've used something like pkg install emacs before.
With commands like that, pkg will connect to the remote repository, ingest the catalog thus figuring out which installed packages have available updates, then it will download the requested package and install it, along with any dependencies that have been updated.

So you've definitely used pkg to update your system before. You just didn't realize what was going on underneath.
In conjunction with synth, the repository pkg(8) connects to is local rather than one of the remote official freebsd ones. But the first time you run synth, the entire repo has to be built up. After that it's usually incremental builds.
 
marino said:
I'm certain you've used something like pkg install emacs before.
Click to expand...

I'm sure I have never used it. I was taught from day one just to use make config, make install clean and portupgrade (later transitioned to portmaster). Also, when I first started using FreeBSD, PKG wasn't an option, it was only PKG_* - I never used those directly either.
 
hmm? taught by whom?
FreeBSD has been a binary package first model for several years now.
Way back when, FreeBSD was known for building everything by scratch, but that's basically discouraged now. The philosophy now is only build if you need non-standard (and thus poorly tested) options, or if you are one of the tin-hat people that think all freebsd packages are compromised by the NSA (or a small 3rd group that think they tailor CPU settings and end up with "better" executables).

the point is that nobody is saying that building from source is the primary way to acquire 3rd party software. The official line is "use packages".

pre-pkg(8) (as you indicate is your generation), it was common to build everything because binary packages had lots of issues. Most of those issues have been eliminated and official packages are trustworthy IMO.
 
A good friend of mine showed me the ropes a long time ago. It just worked for me, so I never looked at anything other method - until now.

I still want to be able to control what gets installed (php extensions, apache modules, etc) plus control the port options myself. I believe Synth is the answer for me, rather then using the official binary packages.

I've always had great dedicated servers, so I never cared about tailoring anything for more speed or better executables.
 
This is really impressive! It only took 21 mins to do all 152 packages. If I remember correctly, it took well over an hour using portmaster.

Also, PKG found a bunch of issues. How it spells everything out is really impressive.

I noticed that is was going to remove MySQL 5.5 client & server, but only install mysql5.6 client - so I need to still figure that out.

Code: 
$ sudo pkg upgrade -n
Updating custom repository catalogue...
custom repository is up to date.
All repositories are up to date.
Checking for upgrades (142 candidates): 100%
Processing candidates (142 candidates): 100%
Checking integrity... done (4 conflicting)
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 13 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        mysql55-server-5.5.56
        mysql55-client-5.5.56
        jpeg-8_7

New packages to be INSTALLED:
        mysql56-client: 5.6.36
        liblz4: 1.7.5,1
        jpeg-turbo: 1.5.1

Installed packages to be REINSTALLED:
        sphinxsearch-2.2.11_1 (direct dependency changed: mysql56-client)
        postfix-3.2.2,1 (direct dependency changed: mysql56-client)
        phpmailer-5.2.23 (direct dependency changed: php71)
        phpMyAdmin-4.7.2 (direct dependency changed: php71-ctype)
        php71-gd-7.1.7 (direct dependency changed: jpeg-turbo)
        gdbm-1.13_1 (needed shared library changed)
        dovecot2-2.2.31_1 (direct dependency changed: mysql56-client)

Number of packages to be removed: 3
Number of packages to be installed: 3
Number of packages to be reinstalled: 7

The operation will free 66 MiB.
 
you have to define the mysql version in the <profile>-make.conf, otherwise it will build the default version of mysql.
There should be lots of posts about defining default versions, and check the handbook (and maybe synth man page)
 
Anything else need to be added for it to work? I just copy & pasted my /etc/make.conf into /usr/local/etc/synth/LiveSystem-make.conf.

I also ran synth rebuild-repository and it still wants to get rid of 5.5 and use 5.6

Code: 
CPUTYPE?=corei7-avx

# /usr/ports/Mk/bsd.options.desc.mk
OPTIONS_UNSET = X11 CUPS NLS

MAKE_JOBS_NUMBER = 8
WITH_POSTFIX = yes
DEFAULT_VERSIONS += ssl=openssl php=7.1 mysql=5.5

WRKDIRPREFIX = /ram
 
big sin, just copying and pasting that.
HUGE ERROR 1: defining MAKE_JOBS_NUMBER. get rid of that line immediately
HUGE ERROR 2: defining WRKDIRPREFIX. get rid of that line immediately
ASKING FOR TROUBLE: defining CPUTYPE. I would remove it

You've just illustrated one good reason why /etc/make.conf is not used and why each profile has a make.conf.
 
Mayhem30 said:
Anything else need to be added for it to work? I just copy & pasted my /etc/make.conf into /usr/local/etc/synth/LiveSystem-make.conf.

I also ran synth rebuild-repository and it still wants to get rid of 5.5 and use 5.6
Click to expand...

Did you actually call synth prepare-system first? Anything that is affected by your make.conf changes has to be rebuilt.
 
Ok, I ran synth prepare-system after correcting the make.conf changes and it shows :

Code: 
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
dovecot2-2.2.31_1.txz failed dependency check.
postfix-3.2.2,1.txz failed dependency check.
sphinxsearch-2.2.11_1.txz failed dependency check.

The task is complete.  Final tally:

Initial queue size: 3
    packages built: 3
           ignored: 0
           skipped: 0
            failed: 0

Duration: 00:01:24

Those are the 3 files that depend on MySQL 5.6. It still wants to use 5.6 though.

Code: 
$ pkg upgrade -n     
Checking for upgrades (142 candidates): 100%
Processing candidates (142 candidates): 100%
Checking integrity... done (4 conflicting)
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 13 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        mysql55-server-5.5.56
        mysql55-client-5.5.56
        jpeg-8_7

New packages to be INSTALLED:
        mysql56-client: 5.6.36
        liblz4: 1.7.5,1
        jpeg-turbo: 1.5.1

Installed packages to be REINSTALLED:
        sphinxsearch-2.2.11_1 (direct dependency changed: mysql56-client)
        postfix-3.2.2,1 (direct dependency changed: mysql56-client)
        phpmailer-5.2.23 (direct dependency changed: php71)
        phpMyAdmin-4.7.2 (direct dependency changed: php71-ctype)
        php71-gd-7.1.7 (direct dependency changed: jpeg-turbo)
        gdbm-1.13_1 (needed shared library changed)
        dovecot2-2.2.31_1 (direct dependency changed: mysql56-client)

Number of packages to be removed: 3
Number of packages to be installed: 3
Number of packages to be reinstalled: 7
 
You could be connecting with the freebsd repository, not the one you built.
try
pkg upgrade -r Synth -n

to explicitly use the local repository
 
Weird .. I thought I had FreeBSD repository disabled. This is what I have done :

/usr/local/etc/pkg/repos/FreeBSD.conf :
Code: 
FreeBSD: { enabled: no }
Also, I had to create my own custom.conf in the repos directory (was Synth supposed to do that for me?)

/usr/local/etc/pkg/repos/Synth.conf :
Code: 
custom: {
  url: "file:///var/synth/live_packages/",
  enabled: yes
}
Using pkg upgrade -r custom -n fixes the issue, but I would like to disable the FreeBSD repo and default it to my custom one.
 
synth creates one, but I think you have to try to install something through synth, e.g. "synth install dovecot" or "synth upgrade-system" . After the first installation, you can use "-r Synth".

It will probably overwrite what you did manually.
 
No way to disable the FreeBSD repo and default it to the Synth repo?

That way I don't have to point to the correct repo every time.
 
You don't need to disable it. Synth version takes precedence. It starts with "00_" and the repo config files are processed in alphabetical order. (so your version WON'T be overwritten).
I don't know if you can actually disable it. I think if nothing is configured it always goes back to the official repo.


added: But I don't trust the precedence so I always specify "-r Synth" to be sure.
 
Ok, synth install databases/phpmyadmin created the missing 00_synth.conf file and everything is working great now.

I checked using pkg -vvv and only the Synth repo is listed at the bottom - which appears the FreeBSD repo was successfully disabled.

Thanks again for all your help (and patience).
 
Does Synth (or PKG) have any commands that will compare local and remote database - and remove any packages from the repository that is not currently installed?

Code: 
$ pkg stats
Local package database:
        Installed packages: 149
        Disk space occupied: 1 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 152
        Unique packages: 152
        Total size of packages: 199 MiB
 
I don't think I understand the question. Or the ultimate purpose.
"remove any packages from the repository".
For most people, there's only one repository, the remote one. Nobody can "remove" packages from the remote repository.
For synth and poudriere users, there is a local repository. Obsolete packages are removed before the repository is built, so everything is current.
Moreover, what's the purpose of "removing packages from a repository"? to save disk space?

The answer is "I don't know of any commands" but as I said, I think I don't even understand what you're getting at.
 
I'm sorry, I should have been more clear in my explanation. I noticed the Synth repository contains 152 items. However, I only have 149 packages installed.

Is there any way to compare the 152 items in the repository with the 149 items I have installed - to let me know which (3 files) can be deleted in repository?

I'm thinking over time that number will grow and would be nice to not have any obsolete items sitting in the repository when they are not needed or being used.
 
You must log in or register to reply here.
Back
Top