Problem updating packages

S

Sam9978

We have multiple servers each running multiple jails. All servers and jails are running fully updated versions of FreeBSD 12.4. We also use Poudriere to create our own packages.

I just recently went to update packages in the jails and most of them update fine but a few refuse to work with the following error

Code: 
Updating packages repository catalogue...
pkg: An error occured while fetching package
pkg: An error occured while fetching package
repository packages has no meta file, using default settings
pkg: An error occured while fetching package
pkg: An error occured while fetching package
Unable to update repository packages
Error updating repositories!

Most posts I've read seem to indicate this is a network issue and in watching the Apache access logs on our package server I can see that there is no request for "meta.conf" and "packagesite.pkg" which made me think that maybe this is a network issue but networking seems fine, I can fetch those files, "host" returns the IP, "drill _https._tcp.packages.example.com SRV" returns the SRV record, etc.

I've also tried using other DNS servers, removing pkg and reinstalling an older version from the backup, running "pkg-static bootstrap -f", restarting the servers and jails, and a few other things but nothing works.

Switching to the FreeBSD repository works but I kind of need to use our repository.

Code: 
Updating FreeBSD repository catalogue...
WARNING: Meta v1 support will be removed in the next version
FreeBSD repository is up to date.
All repositories are up to date.
WARNING: Meta v1 support will be removed in the next version
Updating database digests format: 100%
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
The following 43 package(s) will be affected (of 0 checked):
...

As I said, it's just a few jails that will not work and all the others are fine. There's no discernible difference between any of the jails except for the packages that are installed otherwise they are all set up exactly the same.

I'm going crazy trying to get this to work and would appreciate any ideas anyone has.

Thank you!
 
Have pkg been upgraded lately? I am on 13.2 and recently had the same problem.
The problem was caused by integrating openssl3 into pkg. Take a look at the log from https://www.freshports.org/ports-mgmt/pkg/
Solution was to upgrade to latest pkg.
Temporary removing 'signature_type: "pubkey",' in he repo.conf file also worked.
 
Are you using mirror_type: "srv" and url: "pkg+http[s]:.... in your repository config?

from the changelog of ports_mgmt/pkg:
- improve SRV and HTTP mirroring: reducing the number of round trips
Click to expand...

This change seems to have changed (break?) the behaviour of the "srv" type, as I also was unable to update hosts from my poudriere repositories. removing the mirror_type and "pkg+" from the config solved it for me.
 
BjarneB said:
Have pkg been upgraded lately? I am on 13.2 and recently had the same problem.
The problem was caused by integrating openssl3 into pkg. Take a look at the log from https://www.freshports.org/ports-mgmt/pkg/
Solution was to upgrade to latest pkg.
Temporary removing 'signature_type: "pubkey",' in he repo.conf file also worked.
Click to expand...

Thanks for the reply!

I ran pkg-static bootstrap -f which updated pkg to 1.20.4 which is the latest version but still get the error.
 
sko said:
Are you using mirror_type: "srv" and url: "pkg+http[s]:.... in your repository config?

from the changelog of ports_mgmt/pkg:


This change seems to have changed (break?) the behaviour of the "srv" type, as I also was unable to update hosts from my poudriere repositories. removing the mirror_type and "pkg+" from the config solved it for me.
Click to expand...

Yes, that change definitely broke things for me. The person who set this up used mirror_type: "srv" and url: "pkg+http[s]:.... in the repository config but never added the DNS record so it broke but once I created the DNS record all was fine until a few days later when this issue popped up.

On your recommendation, I tried without pkg+https and mirror_type: "srv". I also tried plain HTTP and even the IP instead of the domain but nothing works.

I just can't figure out why most jails are fine but a small handful are not even though all the jails are essentially the same.

Here's the output of pkg -vvv in case that helps.

Code: 
jail-test:~ # pkg -vvv
Version                 : 1.20.4
PKG_DBDIR = "/var/db/pkg";
PKG_CACHEDIR = "/var/cache/pkg";
PORTSDIR = "/usr/ports";
INDEXDIR = "";
INDEXFILE = "INDEX-12";
HANDLE_RC_SCRIPTS = false;
DEFAULT_ALWAYS_YES = false;
ASSUME_ALWAYS_YES = false;
REPOS_DIR [
    "/etc/pkg/",
    "/usr/local/etc/pkg/repos/",
]
PLIST_KEYWORDS_DIR = "";
SYSLOG = true;
ABI = "FreeBSD:12:amd64";
ALTABI = "freebsd:12:x86:64";
DEVELOPER_MODE = false;
VULNXML_SITE = "[URL]http://vuxml.freebsd.org/freebsd/vuln.xml.xz[/URL]";
FETCH_RETRY = 3;
PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [
]
DEBUG_SCRIPTS = false;
PLUGINS_CONF_DIR = "/usr/local/etc/pkg/";
PERMISSIVE = false;
REPO_AUTOUPDATE = true;
NAMESERVER = "";
HTTP_USER_AGENT = "pkg/1.20.4";
EVENT_PIPE = "";
FETCH_TIMEOUT = 30;
UNSET_TIMESTAMP = false;
SSH_RESTRICT_DIR = "";
PKG_ENV {
}
PKG_SSH_ARGS = "";
DEBUG_LEVEL = 0;
ALIAS {
    all-depends = "query %dn-%dv";
    annotations = "info -A";
    build-depends = "info -qd";
    cinfo = "info -Cx";
    comment = "query -i \"%c\"";
    csearch = "search -Cx";
    desc = "query -i \"%e\"";
    download = "fetch";
    iinfo = "info -ix";
    isearch = "search -ix";
    prime-list = "query -e '%a = 0' '%n'";
    prime-origins = "query -e '%a = 0' '%o'";
    leaf = "query -e '%#r == 0' '%n-%v'";
    list = "info -ql";
    noauto = "query -e '%a == 0' '%n-%v'";
    options = "query -i \"%n - %Ok: %Ov\"";
    origin = "info -qo";
    orphans = "version -vRl?";
    provided-depends = "info -qb";
    rall-depends = "rquery %dn-%dv";
    raw = "info -R";
    rcomment = "rquery -i \"%c\"";
    rdesc = "rquery -i \"%e\"";
    required-depends = "info -qr";
    roptions = "rquery -i \"%n - %Ok: %Ov\"";
    shared-depends = "info -qB";
    show = "info -f -k";
    size = "info -sq";
    unmaintained = "query -e '%m = \"[EMAIL]ports@FreeBSD.org[/EMAIL]\"' '%o (%w)'";
    runmaintained = "rquery -e '%m = \"[EMAIL]ports@FreeBSD.org[/EMAIL]\"' '%o (%w)'";
}
CUDF_SOLVER = "";
SAT_SOLVER = "";
RUN_SCRIPTS = true;
CASE_SENSITIVE_MATCH = false;
LOCK_WAIT = 1;
LOCK_RETRIES = 5;
SQLITE_PROFILE = false;
WORKERS_COUNT = 0;
READ_LOCK = false;
IP_VERSION = 0;
AUTOMERGE = true;
VERSION_SOURCE = "";
CONSERVATIVE_UPGRADE = true;
PKG_CREATE_VERBOSE = false;
AUTOCLEAN = false;
DOT_FILE = "";
REPOSITORIES {
}
VALID_URL_SCHEME [
    "pkg+http",
    "pkg+https",
    "https",
    "http",
    "file",
    "ssh",
    "tcp",
]
ALLOW_BASE_SHLIBS = false;
WARN_SIZE_LIMIT = 1048576;
METALOG = "";
OSVERSION = 1204000;
IGNORE_OSVERSION = false;
BACKUP_LIBRARIES = false;
BACKUP_LIBRARY_PATH = "/usr/local/lib/compat/pkg";
PKG_TRIGGERS_DIR = "/usr/local/share/pkg/triggers";
PKG_TRIGGERS_ENABLE = true;
AUDIT_IGNORE_GLOB [
]
AUDIT_IGNORE_REGEX [
]
COMPRESSION_FORMAT = "";
COMPRESSION_LEVEL = -1;
ARCHIVE_SYMLINK = false;
REPO_ACCEPT_LEGACY_PKG = false;
FILES_IGNORE_GLOB [
]
FILES_IGNORE_REGEX [
]


Repositories:
  packages: {
    url             : "pkg+[URL]http://packages.test.com/12/[/URL]",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV"
  }
 
Last edited by a moderator:
I did a little more testing and pkg-1.19.2 works but 1.20.3 and 1.20.4 do not work. I don't have any versions in between so I can't narrow it down further than that.

I've also tried several different URLs and, with the exception of pkg bootstrap -f, pkg will not even attempt to contact the domain in the config file unless it's one of the "official" FreeBSD servers or mirrors which I find very odd.

I'm going crazy and don't know what to do except recreating all the jails from scratch which I really really don't want to have to do as it would be a very big job.
 
Do you have regular zfs snapshots in place? then you could simply copy pkg or even better pkg-static from the snapshots and pkg lock it until the problem has been resolved e.g. via a test jail.
 
I have exactly the same issue, also relating to hosts and jails running FreeBSD 12.4, also installing packages from a poudriere build.

I think the suggestion from sko is a good one though it requires downgrading ports-mgmt/pkg for all affected machines/jails that can't currently read from my centralised package repository.
 
sko said:
Do you have regular zfs snapshots in place? then you could simply copy pkg or even better pkg-static from the snapshots and pkg lock it until the problem has been resolved e.g. via a test jail.
Click to expand...

We do have zfs snapshots. I actually still have 1.19.2 in /var/cache/pkg so I installed it from there and locked it which worked! pkg upgrade was forcing me to upgrade pkg and I didn't realize pkg lock would get me around that.

I'd still like to figure out the root cause but at least I can upgrade packages now. Thank you for your help!
 
sko said:
Are you using mirror_type: "srv" and url: "pkg+http[s]:.... in your repository config?

from the changelog of ports_mgmt/pkg:


This change seems to have changed (break?) the behaviour of the "srv" type, as I also was unable to update hosts from my poudriere repositories. removing the mirror_type and "pkg+" from the config solved it for me.
Click to expand...
Yes, you will need to create an SRV record or set your mirror type to http. I believe it's always been this way except that it didn't work before.

SRV records are quite handy in that you you can offer your poudriere services on a different port as well, advertising the host and port your service is on. This has been used for kerberos services for ages. For example it's not uncommon to have multiple Kerberos realms served off different ports using different KDCs on the same server, each domain's SRV record pointing to the same server but different port. In this way one doesn't need to update krb5.conf on each client machine. Similarly one could do the same with poudriere, except, why would one when the webserver could serve packages from a different directory. But you get the point, the flexibility can provide solutions to all kinds of weird problems.
 
I'm not using mirror_type: "srv" in my configuration:
Code: 
my_poudriere_repo: {
  url:            "http://user:password@repo_server.local/${VERSION_MAJOR}${VERSION_MINOR}${ARCH}-default"
  enabled:         true,
  signature_type: "pubkey"
  pubkey:         "/etc/ssl/keys/my_poudriere_repo.pub"
  mirror_type:    "none"
}
cy@ is there something you can see wrong with my config?
 
Hi guys,

I usualy used latest for a more up to date version of my packages ... yesterday ... I setup 2 new machines with 13.2 amd64.
One is working fine, the second one is also supporting the same packages ... but couldn't install Firefox... it's not because I have an error message ... it's because it's not in the index .... a search return nothing for Firefox ...
There was only a few hours between the 2 boxes :-( ... I can see that the build date on all mirrors is the 20 of July ... only 2 days ago...

Any one else with missing packages?

Thanks,
Marc
 
I did some more investigation into my issue and believe there may be a bug in ports-mgmt/pkg. I will log a PR unless someone beats me to it.

Version 1.19.2 works just fine and appears to use fetch(3), which I can see when I run in debug mode with pkg -d update.

Version 1.20.4 does not work. Running it in debug mode, it looks like it uses libcurl(3). I set up my web server to use HTTP digest authentication but this version of ports-mgmt/pkg tries to use HTTP basic authentication and does not change mode in response to the server's 401 reply. From the release notes it appears that the switch from fetch(3) to libcurl(3) occurred between versions 1.19.2 and 1.20.0.

I set up with my web server with some rudimentary security mainly so someone couldn't easily see what potentially vulnerable software my machines were running but as this web server can only be accessed from inside my network anyway, there isn't a huge loss to removing it. As I see it my options are:
  • Remove digest authentication for the web server hosting my packages, downgrading to basic or none
  • Lock to a known working ports-mgmt/pkg until the bug is fixed.
I'm inclined to go with the locking option.

Edits: Corrected typos.
 
asteriskross, thank you for reminding me about the debug option I somehow totally forgot about that and it helped me find my issue.

In the jails that don't work the folders /etc/ssl/certs/ and /etc/ssl/blacklisted/ are missing hence the error

Code: 
# pkg -d update
....
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
....

What would be the best way to fix this? Is simply copying those directories from the host server good enough or is there a "proper" way to do it?
 
Last edited by a moderator:
getopt said:
a) Did it fix your problem?

b) See if certctl -v rehash shows more actions taken.
Click to expand...

a) I ran the command last night and didn't think anything of it but today when I tried pkg upgrade I got the original error again so I guess it unfortunately didn't fix the problem. Copying /etc/ssl/certs, /etc/ssl/blacklist, and /usr/share/certs from a working system fixes the issue but certctl rehash doesn't (or if I run it after the copy it causes the problem again).

b)
Code: 
certctl -v rehash
Scanning /usr/local/share/certs for certificates...
Reading ca-root-nss.crt
Adding cd8c0d63.0 to trust store

Thanks for your time, I appreciate it!
 
asteriskRoss said:
I'm not using mirror_type: "srv" in my configuration:
Code: 
my_poudriere_repo: {
  url:            "http://user:password@repo_server.local/${VERSION_MAJOR}${VERSION_MINOR}${ARCH}-default"
  enabled:         true,
  signature_type: "pubkey"
  pubkey:         "/etc/ssl/keys/my_poudriere_repo.pub"
  mirror_type:    "none"
}
cy@ is there something you can see wrong with my config?
Click to expand...
Set mirror_type: "http". mirror_type none is used when your repo is on an NFS share, which BTW I've used years ago.
 
  • Thanks
Reactions: sko
I see many things here:
1/ if you have a custom repository whatever it is always use mirror_type=none and nothing else, mirror_type="http" is a special thing, which makes pkg auto discovers the list of mirrors over http, so unless you know exactly what you do it should not be used.
2/ same goes for mirror_type=srv, to my knowledge only FreeBSD and pfsense are providing mirrors over SRV records, 99.9% of the users hosting their own repo should use mirror_type=none
3/ regarding the issue with the pubkey signature, there is a specific issue here, with openssl3, pkg on freebsd 12.x and 13.x will accept any signature made by pkg repo run on a 12.x, 13.x and 14 version, but pkg current will not accept any repository signed with a pkg binary linked to openssl 1.1, this was due to a misusage of the openssl API by pkg which was accepted by openssl until 3.0, to summerize pkg is backward compatible but not forward compatible.
4/ regarding the basic authent issue I am investigating it right now, so I can't provide an answer yet.
 
Many thanks, bapt@ , the patch is looking good.

Thanks also for clarifying the meaning of MIRROR_TYPE in repository configuration files. IMHO for your next release it would be worth expanding on the MIRROR_TYPE information in the pkg.conf(5) man page to explain when, why and how to use the different values, for example including your points that custom repositories should always use none and that choosing srv requires SRV DNS records.
 
You must log in or register to reply here.
Back
Top