Find out the quartly branch from which packages are installed?

O

outpaddling

I'm looking for a way to automatically install a ports tree that matches the current package repo.

Note for those who are about to say that people should avoid mixing ports and packages:

You can't. There are many ports that cannot be packaged for licensing reasons (audio/lame, sysutils/fusefs-exfat, etc).

So to avoid dependency version mismatches, anyone using binary packages should make sure that their ports tree is the same branch as their pkg repo and update both of them regularly. With quarterly, this will almost guarantee that there are no version mismatches, since versions changes are essentially forbidden. With latest, binary packages may take up to a few days to appear after ports commits, but incompatible versions are extremely unlikely as long as you update both your packages and ports tree regularly.

The problem with what I'm attempting is that the url in pkg.conf simply contains "latest" or "quarterly" rather than the specific quarterly branch such as "2020Q2". In order to check out the proper ports tree, we need to determine exactly which quarterly branch is currently in effect.

I'm wondering if the specific quarterly ports branch from which the package repo is built is even embedded anywhere in local files and easily queried. I'm not seeing anything in the package docs and I've tried poking around /var/db/pkg with sqlite to to avail. I'd rather not try to guess it based on the current date or query the package server to see which branch "quarterly" is linked to.

Thanks,

JB
 
Might be able to do this manually by matching up program versions.
Run pkg info and it will give you the version of every package on your computer
Then go to the package repository for you FreeBSD version (11 or 12) and arch.
You then can match up the package versions from the various repositories.

pkg0.nyi.FreeBSD.org

pkg.freebsd.org
If you notice they use different directories with a numerical value for each quarter (0/1/2/3):
Some programs like youtube-dl will make this very easy because they use the date as the version.
 
outpaddling said:
I'm looking for a way to automatically install a ports tree that matches the current package repo.
Click to expand...
There is currently no such facility available. You need to pull the desired ports tree manually, but you can write a script that does it automatically.

outpaddling said:
The problem with what I'm attempting is that the url in pkg.conf simply contains "latest" or "quarterly" rather than the specific quarterly branch such as "2020Q2". In order to check out the proper ports tree, we need to determine exactly which quarterly branch is currently in effect.
Click to expand...
There are no multiple supported quarterly branches in effect. There are only two supported branches at any one time being: HEAD and Quarterly.
https://wiki.freebsd.org/Ports.

outpaddling said:
I'm wondering if the specific quarterly ports branch from which the package repo is built is even embedded anywhere in local files and easily queried.
Click to expand...
There is no such information embedded somewhere, since it’s assumed and expected that the user uses only a supported branch of the ports tree (or packages repository), or quarterly or HEAD/latest.
 
Thanks for the advice, everyone.

I'm aware that there is only one quarterly branch supported at a given time. That fact does not change the problem, though. An svn update on last quarter's ports tree will not advance it to the current quarter, while pkg update will advance the repo, so we need a way to compare the specific branch tags of both the tree and the pkg repo.

For now I'm just computing the quarter as $(date +'%Y')Q($(date +'%m') - 1) / 3. This is not perfect and will be off between the start date of a quarterly branch and completion of package builds, but that's far better than mixing latest and quarterly. The remaining issue this leaves is that the ports tree will be one quarter ahead of the pkg repo for about a week at the start of each quarter.

I added script, auto-check-ports-branch, to the work-in-progress version of sysutils/auto-admin, so it will be available in the next commit. It will be used by tools like auto-update-system, auto-ports-checkout, and sysutils/desktop-installer to greatly reduce the frequency ports/pkg mismatches.

I also put in a feature request for port-mgmt/pkg so that we might have a clean and reliable solution in the future. If/when that happens, I'll update auto-check-ports-branch.
 
outpaddling said:
For now I'm just computing the quarter as $(date +'%Y')Q($(date +'%m') - 1) / 3. This is not perfect and will be off between the start date of a quarterly branch and completion of package builds, but that's far better than mixing latest and quarterly. The remaining issue this leaves is that the ports tree will be one quarter ahead of the pkg repo for about a week at the start of each quarter.
Click to expand...

Thats what I am doing, in the svn update routine: if the quarter doesnt match the current ports checkout, svn is upgraded to "switch". Also the checkouts are snapshoted as "Cur" and "New", so that the currently installed ports can still be re-built for maintenance, while the new ones are test-built and eventual problems resolved. Then at some point, when everything looks fine, a full pkg upgrade -f is done, and then "New" is renamed to "Cur".

I have no idea if poudriere etc. can do that, my solution is a bunch of shell scripts which grew over the years to the demand (and are now at 3400 loc), which are running along on my desktop, and so I can rebuild everything because with this scheme there is no hurry in it.
 
Right now I'm thinking primarily of simple use cases where binary packages are used wherever possible.

The svn switch idea is useful for this case, though. A quick test showed that switching from 2020Q2 to 2020Q1 takes about 1.5 minutes while a fresh checkout takes 7. Of course it can only be used if both the before and after trees are from svn (vs portsnap or git), but I think it's worth adding a little logic to use it where applicable.

Automatically upgrading ports that cannot be packaged is a separate issue, but shouldn't be too difficult.

Thanks...
 
It occurred to me that another possible solution to this problem would be a ports branch called "quarterly" that's kept precisely in sync with the quarterly packages. I.e. it would switch to a new quarterly branch at the same instant the new quarterly packages go live (not immediately after a new quarterly ports branch is created).
 
Yeah, those are things people can do now, but my goal is to enable a smooth and automatic transition across quarterly boundaries via routine updates. I.e. eliminate the need for manual intervention every 3 months.
 
The following returns the latest quarterly branch of the moment,
Code: 
$curl  https://svnweb.freebsd.org/ports/branches/ | grep /2020Q |sed -n 's/.*href="\([^"]*\).*/\1/p' | sort | grep -v log
 
Thanks, but that's not what we need.

What we need to know is which branch the quarterly binary package repo is currently based on, not what the latest ports branch is. The quarterly package repo will suddenly switch to a new quarterly ports branch when the package farm completes a bulk build for the new quarter. At that moment, every system using quarterly packages needs to switch their ports tree to match to avoid conflicts. We don't currently have a simple and precise way to make this happen.

When exactly the package builds will complete is unpredictable. It depends on the set of packages that need to be rebuilt, the CPU architecture, the current build farm hardware, etc. It's probably just a few days for amd64, but much slower for other architectures that don't have similar hardware resources for the bulk builds.

So we either need a way to detect which ports branch is used by the pkg repo, or a ports branch with a fixed tag (e.g. "quarterly" instead of "2020Q2") that always represents the latest quarter. I think the latter is actually a cleaner solution and should be easy to implement and link to bulk build completion.
 
Does not need to be complicated. When pkg upgrade proposes to upgrade 2500 packages instead of 2, I guess it's a new quarter and time to check out the new ports to remain both in sync.
 
Yes Alain, this is obvious for a human operator, but he wants to automate it, and I think I see his point.
 
You must log in or register to reply here.
Back
Top