Package cache?

[decuser]

I've been tracking discussions about downloading copies of the pkg repos and I'm completely unwilling (unable) to download 130+gigs just to have a local pkg repo. But, if I had limitless bandwidth and faster machines, I'd probably do it in case of zombie apocalypse . I'd like to see if there's another less bandwidth intensive approach...

I'm thinking what if I could just have a copy of my pkgs laying around somewhere after doing pkg install this that and the other that could be used as a cache. That way, if I had several identically provisioned machines (I do), I could just point the others at the "cache". Is this possible?

In TDD style, I'm thinking something along the lines of:

pkg install cu --add-to-cache /pkg-cache

which would install cu and put it and any other things that got installed into /pkg-cache, which could be shared out as odin:/pkg-cache. Then on fenris:

pkg install cu --from-cache odin:/pkg-cache

or somesuch... if this is already supported, point me that way, otherwise, if you can think of a workaround that'd be helpful too.

- will

p.s. It'd be nice if it supported pkg upgrade

 

[SirDice]

You know packages are already cached in /var/cache/pkg/?

 

[decuser]

Supposing I wanted to install from the cached directory, how would I do that? I looked through
man pkg but didn't see anything like:

pkg install --repo /var/cache/pkg/ aria2

Also, the cache has more than just simple pkg files:

/var/cache/pkg/aria2-1.37.0.pkg
/var/cache/pkg/aria2-1.37.0~7a845a00e2.pkg

What's up with that? Is it some sort of checksum.

 

[monwarez]

Pretty sure that if you have /var/cache/pkg mounted on a shared filesystem, it will work as you want. It is in a cache directory after all.
If pkg detect that you already have the package in the cache, it will use the cache, that is the whole point of having a cache in the first place.

 

[Phishfry]

What's up with that? Is it some sort of checksum.

No they changed the way package cache works.
Used to be an archive txz file. Like pkg-name.txz and a un-packed pkg-name.pkg .

Now both files use the same name with *.pkg file extension but the ones with the numbers is the "archive" version.
It is archive file but same extension as extracted *.pkg file.

/var/cache/pkg/aria2-1.37.0.pkg
/var/cache/pkg/aria2-1.37.0~7a845a00e2.pkg

So first file is raw package and second file is the archive file it was extracted from.

 

[Phishfry]

/var/cache/pkg/aria2-1.37.0~7a845a00e2.pkg

I don't know where the suffix comes from for the archive file. I would presume a ports tree branch/tag but I could be wrong.

 

[T-Aoki]

Also, the cache has more than just simple pkg files:

/var/cache/pkg/aria2-1.37.0.pkg
/var/cache/pkg/aria2-1.37.0~7a845a00e2.pkg

What's up with that? Is it some sort of checksum.

The former (without "~7a845a00e2" in its filename) should be a symlink to the latter. Not sure how the additional string is generated, it would be something like hash, encoded timestamp or encoded arch/osversion/branch to determine which should be used if multiple pkg files for the same port is cached.

% ls -l /var/cache/pkg/a*
lrwxr-xr-x  1 root wheel      33  3月  2  2023 /var/cache/pkg/at-spi2-atk-2.38.0.pkg -> at-spi2-atk-2.38.0~cc9f89aa64.pkg
-rw-r--r--  1 root wheel   63592  2月 18  2023 /var/cache/pkg/at-spi2-atk-2.38.0~cc9f89aa64.pkg
lrwxr-xr-x  1 root wheel      34  3月  3  2023 /var/cache/pkg/at-spi2-core-2.44.1.pkg -> at-spi2-core-2.44.1~6692d26bdb.pkg
-rw-r--r--  1 root wheel  205380  2月 18  2023 /var/cache/pkg/at-spi2-core-2.44.1~6692d26bdb.pkg
lrwxr-xr-x  1 root wheel      25  3月  2  2023 /var/cache/pkg/atk-2.38.0.pkg -> atk-2.38.0~deb9530ccf.pkg
-rw-r--r--  1 root wheel  303052  2月 18  2023 /var/cache/pkg/atk-2.38.0~deb9530ccf.pkg
lrwxr-xr-x  1 root wheel      40  5月 18 05:39 /var/cache/pkg/audacious-plugins-4.3.1_4.pkg -> audacious-plugins-4.3.1_4~7be2afe66a.pkg
-rw-r--r--  1 root wheel 1351912  5月 13 17:59 /var/cache/pkg/audacious-plugins-4.3.1_4~7be2afe66a.pkg
lrwxr-xr-x  1 root wheel      30  7月 31  2023 /var/cache/pkg/automake-1.16.5.pkg -> automake-1.16.5~b5a4faa53e.pkg
-rw-r--r--  1 root wheel  563672  7月 31  2023 /var/cache/pkg/automake-1.16.5~b5a4faa53e.pkg

 

[T-Aoki]

man pkg but didn't see anything like:

Use pkg help <subcommand> instead, like pkg help install or pkg help add.
And if you want to newly install a pkg with specific file, use pkg add instead.
It accepts path to the pkg file as argument like pkg add /usr/ports/packages/All/libreoffice-7.6.4.1_2.pkg (old example from my zsh history, so shouldn't work with version mismatch as is).

 

[Cath O'Deray]

SHA256

 

[Cath O'Deray]

… Not sure how the additional string is generated, it would be something like hash, …

Short hash as documented below. SHA256 as pictured above.

% man -P cat 8 pkg-repo | grep -B 5 -A 2 symlink

ENVIRONMENT
     PKG_REPO_HASH     When set, rename packages with the short hash of
                       contents appended to the filename.

     PKG_REPO_SYMLINK  When set, create a symlink between the short hash
                       filename and the regular filename.

%

 

[Andriy]

And if you want to newly install a pkg with specific file, use pkg add instead.

I was surprised to discover, very recently, that pkg install also supported that mode of operation.
I have pkg version 1.21.3 here.
The mode is not documented.

# pkg install /var/cache/pkg/thunderbird-128.2.0.pkg
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        thunderbird: 128.2.0 [unknown-repository]

Number of packages to be installed: 1

The process will require 283 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing thunderbird-128.2.0...
Extracting thunderbird-128.2.0: 100%
==> Running trigger: desktop-file-utils.ucl
Building cache database of MIME types

 

[T-Aoki]

I was surprised to discover, very recently, that pkg install also supported that mode of operation.
I have pkg version 1.21.3 here.
The mode is not documented.

Thanks! I didn't know that, as at the moment I've tried it didn't work (but pkg add worked) and still undocumented as you mentioned.

 

[Cath O'Deray]

not documented

Your local file:

/var/cache/pkg/thunderbird-128.2.0.pkg

That is, a type of archive.

Here, for example:

% file /usr/local/poudriere/data/cache/main-default/zstd-1.5.6.pkg/1722476141/pkg%origin
/usr/local/poudriere/data/cache/main-default/zstd-1.5.6.pkg/1722476141/pkg%origin: ASCII text
% cat /usr/local/poudriere/data/cache/main-default/zstd-1.5.6.pkg/1722476141/pkg%origin
archivers/zstd
% pkg fetch --quiet --yes --no-repo-update --output /tmp archivers/zstd
% file /tmp/All/zstd-1.5.6.pkg
/tmp/All/zstd-1.5.6.pkg: Zstandard compressed data (v0.8+), Dictionary ID: None
% rm -r /tmp/All
%

 

[Andriy]

Cath O'Deray yeah, but the actual usages do not covert that.

 

[Cath O'Deray]

pkg-add(8), pkg-install(8)​

---

2014:

---

2022:

---

Incidentally, re: security,

 

[Andriy]

Just in case, by "actual usages" I meant usages shown in the manual page.

 

[Cath O'Deray]

… by "actual usages" I meant usages shown in the manual page.

?

An EXAMPLES section would be nice in pkg-install(8) … for now, there's the simpler PR for pkg(8). Get consensus on phraseology and concepts before fleshing things out with examples. Thanks.

 

[Mirror176]

Probably should alter its manpage "<pkg-origin|pkg-name|pkg-name-version>" entry to also include filename and clarify if a full path is required if it doesn't support just running that on a file in a local directory by file name. Always good to see examples covering how to do things when its not thought to be the normal way to do it.