Problems with local pkg repo catalog

[ssw01]

I recently did a poudriere run on a 13.0-RELEASE-p3 system to build a couple of ports.
I expected the newly built packages to be added to the corresponding repository because I could've
sworn that was my past experience. Instead, running pkg install <some pkg> for
"some pkg" already in the repo produced a message that "some pkg" wasn't in the repo.
The packagesite.txz seemed to contain only the most recently built couple of
packages.

So I decided I needed to regenerate the catalog and ran pkg repo <repo path>
which produced the output

pkg: No package files have been found
Cannot create repository catalogue

The version of pkg on this machine is 1.17.1.

I tried running the same pkg repo on a 12.2 machine with pkg 1.15.10 specifying the
same repo tree via NFS and it seemed to work fine.

Does this call for a bug report or am I doing something wrong?

----
Edit:

So I force installed pkg-1.16.3 with pkg add -f <path to pkg-1.16.3.txz> and tried pkg repo <repo path>
again which succeeded.

Running pkg install apache24 then worked after reinstalling pkg-1.17.1. So I'm back in business but this
seems peculiar.

 

[Alain De Vos]

Two files which might shed a light

/etc/pkg/FreeBSD.conf

And the contents of

/usr/local/etc/pkg/repos

 

[ssw01]

Here they are. These have been working fine for ages.

Thanks!

 

[Alain De Vos]

If i'm correct your directory,
/var/poudriere/data/packages/r130amd64-muqchem-default
should contain:

meta.conf
meta.txz
packagesite.txz
[ And directories:]
All/
Latest/

 

[ssw01]

It does indeed, although I won't have access to the machine involved until tomorrow so can't
provide an actual listing at the moment. Recently, files of the form packagesite.pkg
which are links to or from the corresponding .txz files have appeared as well.

I unpacked packagesite.txz and looked at the yaml file. It contained only the most recently built
packages which brings up the first part of my question. Suppose I run this sequence:

poudriere bulk -j <jailname> -f <file containing list of ports>
poudriere bulk -j <jailname> <origin1/port1> <origin2/port2>

Would the corresponding repository be expected to contain the union of what's in the
file from the first invocation and the two packages from the second or just the latter?

I could have sworn it was the union but maybe I dreamed that. The actual
contents of <repo root>/.latest/All include the union of all the .txz files
but the metadata seems to have other ideas.

In any case, running pkg repo <path> with rev 1.17.1 fails while the
same command on the same directory using rev 1.16.3 works, so that seems
like a problem.

Thanks!

 

[SirDice]

Some things changed with pkg(8) version 1.17.0. You need to clean out your repository to get rid of the old packages. These seem to get a preference over the newer packages. Just run poudriere pkgclean -j <jailname> -f <file containing list of ports> to have it remove all the old packages.

 

[ssw01]

Thanks SirDice,

I'm a bit confused. The man page for poudriere-pkgclean(8) says the -f option
specifies a list of ports to keep. When I ran poudriere pkgclean -j <jail> -f <my port list>,
poudriere(8) reported that it wanted to remove 1304 ports it had recently
spent a week building. So it looks as if it considers everything I built using the -f option
in the example in message #5 to be stale.

Do I really have to do this to get things back in sync?

 

[SirDice]

The man page for poudriere-pkgclean(8) says the -f option
specifies a list of ports to keep.

That is correct.

When I ran poudriere pkgclean -j <jail> -f <my port list>,
poudriere(8) reported that it wanted to remove 1304 ports it had recently
spent a week building.

Look closely, those are old versions. You can collect a whole bunch of old packages if you continuously update the builds. The repository will keep all those old versions around. So after a while you have multiple (old) versions of various packages. poudriere-pkgclean(8) will clean those out.

 

[ssw01]

Here's the output of ls -l /var/poudriere/data/packages/r130amd64-std-default/.latest/All | tail -5:

-rw-r--r--  1 nobody  wheel     227392 Jul 28 17:23 zip-3.0_1.txz
-rw-r--r--  1 nobody  wheel    4616668 Jul 31 21:27 zsh-5.8.txz
-rw-r--r--  1 nobody  wheel     595452 Aug  2 15:55 zstd-1.5.0.pkg
lrwxr-xr-x  1 nobody  wheel         14 Aug  2 15:55 zstd-1.5.0.txz -> zstd-1.5.0.pkg
-rw-r--r--  1 nobody  wheel     106176 Jul 27 07:37 zziplib-0.13.72_1.txz

Here are the final few lines of output from poudriere pkgclean -j r130amd64-std -f /var/poudriere/portlist-std.txt:

/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zip-3.0_1.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zsh-5.8.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zstd-1.5.0.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zziplib-0.13.72_1.txz
[00:01:34] Removing these 1304 stale packages will free: 3.50 GiB
[00:01:34] Proceed? [y/N] n
[00:03:25] Cleaning up
[00:03:25] Unmounting file systems

This was actually a pretty new repo without much cruft because I created it fresh after updating to 13.

 

[SirDice]

Don't use dashes (-) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.

 

[ssw01]

Oooh, never knew that. Thanks!

 

[SirDice]

Oooh, never knew that. Thanks!

Me neither when I first started with poudriere. It's not in the manual either, but I noticed some really weird behavior if you use them. It might be the reason why poudriere-pkgclean(8) wants to remove perfectly good packages in your case.

 

[ssw01]

OK, since I seem to be able to make the existing repos usable by running pkg repo with a pre-1.17
version of pkg(8), I'll limp along that way until the time comes for a ports update. I'll then create shiny
new jails following your naming advice and rebuild everything.

Thanks again.

 

[Jose]

Don't use dashes (-) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.

Doc bug?

 

[Alain De Vos]

Poudriere puts itself a "-" between the jailname and portsname.

 

[SirDice]

Poudriere puts itself a "-" between the jailname and portsname.

Yeah, that's why I think having a dash in the jail name or ports tree causes a failure to properly identify the jail or ports tree. Options didn't get properly applied, make.conf files didn't work and a whole bunch of other weird behavior.

 

[grahamperrin]

Doc bug?

poudriere naming conventions for jails and ports trees · Issue #897 · freebsd/poudriere

Prerequisites ☑ Have you checked for an existing issue describing the issue? From https://forums.FreeBSD.org/threads/81582/post-525675: Don't use dashes (-) in the name of the jail … or the ports t...

github.com

 

[brd@]

Don't use dashes (-) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.

I have been using dashes in my jail names for many years and have had no issues. If you have found a bug please open an issue and we would be happy to fix it.

 

[brd@]

OK, since I seem to be able to make the existing repos usable by running pkg repo with a pre-1.17
version of pkg(8), I'll limp along that way until the time comes for a ports update. I'll then create shiny
new jails following your naming advice and rebuild everything.

Thanks again.

What version of poudriere are you using?

 

[grahamperrin]

With poudriere-devel-3.3.99.20210720 on FreeBSD 14.0-CURRENT,

root@mowa219-gjp4-zbook-freebsd:~ # grep -v \# /usr/local/etc/poudriere.d/make.conf
WITH_DEBUG_PORTS += multimedia/webcamd sysutils/bsdisks

ICA_CERTS=QuoVadisEuropeEVSSLCAG1.crt

LICENSES_ACCEPTED += commercial

WITHOUT_LLVM_TARGET_ALL=
root@mowa219-gjp4-zbook-freebsd:~ #

With - in the name of a jail:

1. poudriere jail -c -j 13-0-release -v 13.0-RELEASE
2. poudriere jail -u -j 13-0-release
3. poudriere bulk -b latest -j 13-0-release -Ct sysutils/lsblk
4. poudriere bulk -b latest -j 13-0-release -Ct multimedia/webcamd sysutils/bsdisks

From the manifest of the resulting package for webcamd:

"options":{"COMPAT32":"on","DEBUG":"off","DVB":"on","INPUT":"on","KEYBOARD":"off","MOUSE":"off","RADIO":"on","V4L2LOOPBACK":"on","VT_CLIENT":"off","VT_SERVER":"off","WEBCAM":"on"}

– debug off. PEBKAM? Something wrong with my /usr/local/etc/poudriere.d/make.conf maybe?

With my usual jail (for CURRENT) I ran:

• poudriere bulk -b latest -j main -Ct multimedia/webcamd

– then installed the package and:

root@mowa219-gjp4-zbook-freebsd:~ # pkg info webcamd | grep -i debug
        DEBUG          : off
root@mowa219-gjp4-zbook-freebsd:~ #

– again, off but I want it on. What am I missing?

 

[ssw01]

What version of poudriere are you using?

poudriere-devel-3.3.99.20210521.

I've deleted my old jails and created a new one with no '-' in the name and am in the
middle of a build. It has a few days yet to run, so I won't know for a while whether
the behavior is different.

 

[ssw01]

Well, my build finished early because lang/rust failed which caused a bunch of stuff to get skipped but that's neither
here nor there for present purposes. The build command was poudriere bulk -f /var/poudriere/plists/full.txt -j r130amd64full.

I then ran poudriere pkgclean -f /var/poudriere/plists/full.txt -j r130amd64full to see what would happen. Here's the
tail end of the output:

/var/poudriere/data/packages/r130amd64full-default/.latest/All/zip-3.0_1.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zsh-5.8.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zstd-1.5.0.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zziplib-0.13.72_1.txz
[00:01:38] Removing these 1153 stale packages will free: 
[00:01:38] Proceed? [y/N] n
[00:02:08] Cleaning up
[00:02:08] Unmounting file systems

I then ran pkg upgrade ports-mgmt/poudriere-devel to see what upgrading to 20210720 would do. The output:

Updating local repository catalogue...
pkg: Repository local has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.pkg: 100%  276 KiB 282.2kB/s    00:01    
Processing entries: 100%
local repository update completed. 1153 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
    ca_root_nss: 3.63 -> 3.68
    poudriere-devel: 3.3.99.20210521 -> 3.3.99.20210720

Number of packages to be upgraded: 2

Proceed with this action? [y/N]: n

So given the message about "wrong packagesite", I ran pkg repo /var/poudriere/data/packages/r130amd64full-default,
which produced:

pkg: No package files have been found
Cannot create repository catalogue

Also:

> ls -al /var/poudriere/data/packages/r130amd64full-default
total 12
drwxr-xr-x  3 root  wheel  512 Aug 11 08:36 .
drwxr-xr-x  9 root  wheel  512 Aug  6 15:02 ..
lrwxr-xr-x  1 root  wheel   18 Aug 11 01:38 .buildname -> .latest/.buildname
lrwxr-xr-x  1 root  wheel   20 Aug 11 01:38 .jailversion -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 .latest -> .real_1628660288
drwxr-xr-x  4 root  wheel  512 Aug 11 01:38 .real_1628660288
lrwxr-xr-x  1 root  wheel   11 Aug 11 01:38 All -> .latest/All
lrwxr-xr-x  1 root  wheel   14 Aug 11 01:38 Latest -> .latest/Latest
lrwxr-xr-x  1 root  wheel   17 Aug 11 01:38 meta.conf -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 meta.pkg -> .latest/meta.pkg
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 meta.txz -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel   23 Aug 11 01:38 packagesite.pkg -> .latest/packagesite.pkg
lrwxr-xr-x  1 root  wheel   23 Aug 11 01:38 packagesite.txz -> .latest/packagesite.txz
-rw-r--r--  1 root  wheel    0 Aug 11 08:36 packagesite.yaml

Anything else I can try to help track this down?

 

[SirDice]

Updating local repository catalogue... 
pkg: Repository local has a wrong packagesite, need to re-create database

You get that message if the URL to your repository is changed.

 

[ssw01]

You get that message if the URL to your repository is changed.

Ah, gotcha.

So I went ahead and ran the pkg upgrade ports-mgmt/poudriere-devel, so now running
poudriere-devel-3.3.99.20210720.

Then tried the poudriere pkgclean again and got the same result (it wants to delete the
entire repository).

 

[SirDice]

Then tried the poudriere pkgclean again and got the same result (it wants to delete the entire repository).

That's odd as it shouldn't do that, poudriere-pkgclean(8) works as expected for me. I do have a similar issue with poudriere-logclean(8). If I run poudriere logclean 14 it always want to delete all build logs when it should only delete the ones older than 14 days.