Synth: Introducing new custom package repository builder for FreeBSD and DragonFly

The "safe" would be to build everything, then rebuild the repository when everything succeeds.

As I said, I now feel like the convenience commands were a mistake on my part. The consequences of misunderstanding what's happening (which is easy to do since pkg(8) is kind of opaque) can be significant.
 
I was running synth upgrade-system at night until the point a package failed to build, and when I waked up pkg(8) was removed half of my installed packages.

Since then I use synth prepare-system to build the upgrades and then run pkg upgrade manually to upgrade the system, as I can see what will happen.
 
Is it normal for synth to get into a state where it is only running one builder? It started with six, but five are now idle.

Since taking the screenshot, below, the job finished and a new one has started on ID 3. IDs 4 and 6 are showing 'Shutdown'.
 

Attachments

  • Screenshot-2017-05-27-20-58-01.png
    Screenshot-2017-05-27-20-58-01.png
    158.8 KB · Views: 494
Yes, this situation was already discussed here recently.

Basically, when the port being built is a direct or indirect dependency to all other ports, they will wait that one to finish before grab a slot. The slot will be in "Idle" state.

When a slot will not be necessary anymore, due to the dependencies correlation, synth will be in "Shutdown" state.
 
Firstly - many thanks to Marino for Synth, which is excellent.

I have got synth working on my development machine, and can use its repository to update packages on a VM on my laptop. On my development machine I have a couple of jails which I installed with ezjail. These require a base jail, which has its own ports tree. I would like to move to updating the jails using the Synth repository, and avoid using ports with jails. Is this possible?
 
yes. A repository is designed for more than one machine. Your development machine can build a single repository that it, and both jails use. You just put the repository in a place that the jail's pkg(8) can access, and update the pkg/repos conf files to find the repository.
 
Thanks marino. I am already using the repository for two machines. It was the template jail ports stuff that was confusing me a little. Basically, I can just ignore that, and edit /etc/pkg/FreeBSD.conf in each jail to point to my little webserver which serves up the files. I went ahead and tried it, and it just worked.
 
jdp_03 said:
Basically, I can just ignore that, and edit /etc/pkg/FreeBSD.conf in each jail to point to my little webserver which serves up the files.
Click to expand...
Don't edit that file, it can and will be overwritten during an update. Instead create files in /usr/local/etc/pkg/repos/.

Disable the FreeBSD repository with FreeBSD.conf:
Code: 
FreeBSD: { enabled: no }
Then add your own repositories to /usr/local/etc/pkg/repos/. That way they'll never be modified or overwritten during an update/upgrade.
 
I am now a hostage of synth: it screwed something up, so I can't use portmaster anymore as it instantly quits with "===>>> Gathering distinfo list for installed ports ===>>> Starting check of installed ports for available updates ===>>> No origin available for devel/gmake-4.2.1 ===>>> Cannot continue ===>>> Aborting update".
How to fix this issue? synth brings more anger than satisfaction and I'd like to replace it.
 
remove /usr/ports and reinstall it. The directory is corrupt. Portsnap does that sometimes. It's probably giving postmaster problems too.
 
I'm pretty new to synth. Seems like a very nifty tool. I've just installed it on a host, checked synth configure and did a # synth upgrade-system after syncing the ports tree. A lot of things got rebuild and apparently reinstalled as the ports version. So that went well, but now I'm trying to install a bit of Python software and it states it cannot find the package. Probably because the name changes as it becomes a package. I'll show you:


# synth install archivers/py-borgbackup
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100% 260 B 0.3kB/s 00:01
Fetching packagesite.txz: 100% 29 KiB 29.8kB/s 00:01
Processing entries: 100%
Synth repository update completed. 85 packages processed.
All repositories are up to date.
pkg: No packages available to install matching 'archivers/py-borgbackup' have been found in the repositories
Unfortunately, the system upgraded failed.


This is after an earlier try, so the dependencies have already been build.
But my question is how should I install archivers/py-borgbackup ?
Because as package it probably becomes archivers/py36-borgbackup and pkg would be unable to find it as py-borgbackup?
 
archivers/py-borgbackup is definitely in the ports tree, however I could not find it at the FreeBD pkg repository (nothing called borgbackup is in there indeed). You can look at /var/log/synth to see build logs and discover what happened.

Alternativelly, you can look for it at your local synth repository using the pkg search function.
 
To understand what is happening, you have to understand what the commands you use actually do.

As an aside: Instead of using synth upgrade-system, instead use synth prepare-system with package upgrade

so what is synth upgrade-system doing?
The normal operation of synth is to take a list of ports to build. This command says "get that list of ports from pkg(8) by querying what packages are already installed". A better way is to maintain a list of ports as a separate file and just feed that file to synth. However, nobody has such a list at first (although pkg(8) can be used to generate it off course).

What's going on is that that the port of an installed package got renamed. The list that pkg(8) creates is in error because the installed py-borgbackup has the old port location and that's what is getting returned. The bad list is passed to synth.

The fix?
  • Dump the list yourself with pkg prime-origins (or pkg query -e '%a = 0' '%o'), edit it and feed it to synth
  • or remove py-borgbackup package, build it manually via synth, and install it via synth repository.
In any case you'll have to remove the old py-borgbackup. I'm not sure pkg(8) is smart enough to follow the rename.
 
Thanks for your replies.

lebarondemerde said:
archivers/py-borgbackup is definitely in the ports tree, however I could not find it at the FreeBD pkg repository.
Click to expand...
Yeah, it's only available as a port. That's one of the reasons why I changed to all ports instead of packages on this system. Didn't want to mix and match pkgs and ports together, plu I gain customization.
lebarondemerde said:
Alternativelly, you can look for it at your local synth repository using the pkg search function.
Click to expand...
I'll have synth build the port ans see if pkg finds it.

marino said:
As an aside: Instead of using synth upgrade-system, instead use synth prepare-system with package upgrade
Click to expand...
Thanks, I just tried it and it works. I'll change to doing my updates like that in the future.

marino said:
so what is synth upgrade-system doing?
The normal operation of synth is to take a list of ports to build. This command says "get that list of ports from pkg(8) by querying what packages are already installed". A better way is to maintain a list of ports as a separate file and just feed that file to synth. However, nobody has such a list at first (although pkg(8) can be used to generate it off course).

What's going on is that that the port of an installed package got renamed. The list that pkg(8) creates is in error because the installed py-borgbackup has the old port location and that's what is getting returned. The bad list is passed to synth.

The fix?
  • Dump the list yourself with pkg prime-origins (or pkg query -e '%a = 0' '%o'), edit it and feed it to synth
  • or remove py-borgbackup package, build it manually via synth, and install it via synth repository.
In any case you'll have to remove the old py-borgbackup. I'm not sure pkg(8) is smart enough to follow the rename.
Click to expand...
Yeah, I had figured out the rename caused the trouble. There will probably be more ports with the same issue out there.

Seeing as how py[36]-borgbackup isn't installed right now I instructed # synth just-build archivers/py-borgbackup. But it determined there are no packages that require rebuilding. So no package py-borgbackup got built.
I checked out /var/synth/live_packages/All/ and there I don't have it there either. Does look like all the dependencies are still there from the previously failed "install".

I would appreciate some more hints in getting the port archivers/py-borgbackup installed through synth. If you require more info from my end let me know and I'll provide.
(Might be a while before I reply because I'll be going on holiday soon.)
 
stejoo said:
I would appreciate some more hints in getting the port archivers/py-borgbackup installed through synth.
Click to expand...

Sure, here ya go: don't bother. Borg is a Python 3 application. The only way to get Borg to build with either Synth or Poudriere is to change the system-wide default Python version to Python 3, and changing the default Python version has the potential to break any Python 2 packages you might use. So while archivers/borg-backup might build after changing the default, other Python packages might not. That's why it's only available as a port.

If Borg is the only Python application you use, then changing the default is no problem; if you use other Python applications, you're probably stuck. I personally just use pre-compiled binary the project offers. You can also install devel/py-pip and install Borg that way.
 
I re-read your original post. It seems that your complaint is with portmaster.
You also say synth "screwed something up" which makes little sense. Synth is just a repository builder. All it does is build packages for pkg(8). It doesn't have the capability to effect portmaster.

I originally read your post as Synth was having an issue as well as portmaster, but if it's only portmaster with the issue, that's out of scope for me. I have no interest in solving that problem.
 
Hello,

First of all, many thanks to marino for Synth! I started using it a couple of weeks ago, and it just perfectly fits my needs.

I'm having a bit of an issue with updating of readline package (recent update from 6.3.8 to 7.0.3). Actually, I believe my issue has nothing to do with Synth, but pkg, however I'd appreciate any hints.

I'm using Synth 1.69 on FreeBSD 11.1-BETA3 (r320268) to build packages for ~20 VMs running FreeBSD.

So I get a list of all packages installed on VMs, feed to synth, which builds and produces txz files, which are then exposed via http to VMs. Everything is working/building just fine (I think).

Here is the content of /usr/local/etc/pkg/repos/FreeBSD.conf file from one of the VMs:

###
FreeBSD: { enabled: no }

Synth: {
url: "pkg+http://repos.local.net/pkg/",
mirror_type: "srv",
signature_type: "none",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}
###

I have a server with readline-6.3.8_1 installed, which I expected to be updated to 7.0.3 when I execute 'pkg upgrade -r Synth' on that server, however pkg says everything is up to date:

Updating Synth repository catalogue...
Synth repository is up to date.
All repositories are up to date.
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

I confirm that readline-7.0.3.txz was sucessfully built by Synth, and it's indeed present under repos.local.net/pkg/All. Web server logs also confirm that the VM does contact the repository:

192.168.10.33 repos.local.net - [29/Jun/2017:13:11:32 +0200] "GET /pkg//meta.txz HTTP/1.1" 304 0 "-" "pkg/1.10.1"
192.168.10.33 repos.local.net - [29/Jun/2017:13:11:32 +0200] "GET /pkg//packagesite.txz HTTP/1.1" 304 0 "-" "pkg/1.10.1"

However, no update is triggered, or meta.txz (packagesite.txz?) somehow doesn't reflect the change of versions?

Thank you.
 
Hi!

I am experiencing a weird behavior I do not know if related with ports-mgmt/synth or something else. It is with sysutils/fusefs-simple-mtpfs, what should install a file called special_simple-mtpfs in /etc/autofs, what is not happening.

The Makefile show what seems to be some "glue" to make that file to be installed in there, and a comment point to PR 193596:

Code: 
.if exists(/etc/autofs)
PLIST_FILES+=    /etc/autofs/special_${PORTNAME}
SUB_FILES+=    special_${PORTNAME}
.endif

post-install:
    (cd ${WRKSRC} && ${COPYTREE_SHARE} \
        "${PORTDOCS}" ${STAGEDIR}${DOCSDIR})
.if exists(/etc/autofs)
    @${MKDIR} ${STAGEDIR}/etc/autofs
    ${INSTALL_SCRIPT} ${WRKDIR}/special_${PORTNAME} \
        ${STAGEDIR}/etc/autofs
.endif

I had not tried using make directly, however the package (from the FreeBSD repository) install it correctly, and so I am assuming there is no problem with ports-mgmt/poudriere.

Thanks!
 
what that port doing is bogus.
It's tailoring the port to the local machine.
if autofs is needed, the port should install it, not adjust to its possible presence.

if you want to fake synth out, copy /etc/autofs to a temp location that synth uses, e.g. /usr/bin/autofs and modify the makefile accordingly.

I'm not recommending it, just saying how it could be possible to work around this extremely bad practice the port is doing.
 
I will install from packages and copy that file to somewhere else , then I will install from the ports-mgmt/synth repository and copy that file back to /etc/autofs to see if it works this way.

If it works I think it would be better than mess with the Makefile.

Thank you marino.

EDIT: it worked. :cool:
 
You must log in or register to reply here.
Back
Top