pkg repo

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I'm thinking about setting up my own local pkg repository and was advised to look at pkg-repo() but am not very clear about how it works. Do you need to assemble a load of pkgs first in somewhere like /var/cache/pkg and then run this utility? What does it output?
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

So when I've run pkg-repo() what do I need to do to make pkg install try to retrieve pkgs from this repository?

I saw a post which suggested using a web server to make pkgs accessible but not sure if that is the way to do it...
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,626
Messages: 37,981

You need to set up a custom repository in /usr/local/etc/pkg/repos/. While a web server is certainly the easiest solution if you have more hosts on your network, you can actually use any kind of URL, even file://:
Code:
my-repo: {
  enabled: yes
  url: file:///some/local/directory
}
See pkg.conf(5).
 

jmos

Active Member

Reaction score: 176
Messages: 243

Basically you can do this:

Create all packages you've got installed on your package server with pkg create -a inside the desired directory (unfortunately you'll get all packages in one directory). Root shouldn't be necessary, but some backages will fail without…

Afterwards run pkg repo path/to/that/dir/ - your meta files will be created.

Then set up a simple webserver, f.e. lighttpd: You configuration could look like this:

Code:
server.document-root = "/path/to/dir"
server.port = 1120
server.dir-listing = "enable"
mimetype.assign = ()

Note the high port number I've used that makes it possible to run the server without root. Now start your webserver: lighttpd -f configfile.conf

(The last weeks I began writing and testing a simple, but more comfortable tool to turn a given machine into a package server as easy as possible and with low dependencies.)
 
Last edited:
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

You need to set up a custom repository in /usr/local/etc/pkg/repos/. While a web server is certainly the easiest solution if you have more hosts on your network, you can actually use any kind of URL, even file://:
Code:
my-repo: {
  enabled: yes
  url: file:///some/local/directory
}
See pkg.conf(5).
Once I've done that, is it a matter of imply pkg install pkgname and pkg will be directed by via the directive in /usr/local/etc/pkg.conf...?

Not entirely sure where your my_repo is located or how pkg finds it... or is your example part of your pkg.conf?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,626
Messages: 37,981

s it a matter of imply pkg install pkgname and pkg will be directed by via the directive in /usr/local/etc/pkg.conf...?
Put your custom repos in /usr/local/etc/pkg/repos/<repo>.conf. Then check with pkg -vv:
Code:
Repositories:
  dicelan-server: {
    url             : "file:///usr/local/poudriere/data/packages/13-stable-server/",
    enabled         : yes,
    priority        : 0
  }
Disable the FreeBSD repo by creating a /usr/local/etc/pkg/repos/FreeBSD.conf:
Code:
FreeBSD: {
  enabled: no
}
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I've created a test-repo.conf and copied /var/cache/pkg/* into it just to get a feel for how this works, and now when I attempt something like pkg search lfm I get:-

Code:
pkg: Repository test-repo missing. 'pkg update' required
pkg: wrong architecture: FreeBSD:11:amd64 instead of FreeBSD:13:amd64
pkg: repository test-repo contains packages with wrong ABI: FreeBSD:11:amd64

Do I need to sort out these pkgs into different repos? How does FreeBSD handle different ABI's?
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I just copied /var/cache/pkg/* from some system to it. Don't really know how a repo is supposed to look....
 

Alain De Vos

Daemon

Reaction score: 549
Messages: 1,898

There is a certain structure.
It should contain directories All,Latest
It should countain meta.conf, meta.txz, packagesite.txz.
Why and how I don't know. Hope to find out.
meta.conf contains how it is packaged.
packagesite.txz is a large yaml of all the packages available in the repo.
It contains general fields like, name,size, etc ...
The packages are stored as .txz file in directory All.
Copying /var/cache/pkg is not enough. You miss the meta information.
It's like a reduced version of only the directory All

Don't you need compat for different abi's ?
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I don't know how all this is structured... just learning...

pkg-repo() creates the meta.conf, meta.txz packagesite.txz, all of which I have, but obviously something is missing.
 

ShelLuser

Son of Beastie

Reaction score: 2,082
Messages: 3,780

A really easy ("quick and dirty") way to set up a repository would be....
  1. # pkg create -ao /usr/ports/packages/All
  2. openssl genpkey -algorithm RSA -out private.key
  3. openssl pkey -in private.key -out public.key -pubout
  4. # pkg repo /usr/ports/packages private.key
(as a sidenote... openssl isn't necessarily getting better; in the 'old days' you could create both a private and public key in one go)

Anyway... this would grab all your installed packages and publish them into /usr/ports/packages/All. Then you create the repository manifest and sign that with your RSA key.

The next step would be to copy your public key onto the other server(s) which need to use your repo and then inform them about the repo and specify the key (using "PUBKEY" in your config file).

As said: this is a very bare and "dirty" way, but it'd work.[/file]
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

This doesn't sound like 'bare minimum' to me :). I'm using
signature_type: "none",
and don't need any keys, that just gets in the way for me.


I don't really see what I've done differently apart from not creating the All directory. I'm not building any pkgs myself, I just want a single local repository on my LAN, so I don't need to keep fetching the same pkgs off the Internet.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,626
Messages: 37,981

I don't really see what I've done differently apart from not creating the All directory.
You still need that directory structure. The packages should go in an All directory.

Code:
All/  # Contains the packages
meta.conf
meta.txz
packagesite.txz
 

jmos

Active Member

Reaction score: 176
Messages: 243

You still need that directory structure. The packages should go in an All directory.
Package file can be in structured in subdirectories - the pkg-repo command crawls through them, and adds the paths to the meta files. I'm splitting my packages in subdirs as the ports tree does, and: It works. There's no need for an "ALL" directory, you can use whatever you want. Example:

Code:
jo@freya ~>  mkdir mypackages
jo@freya ~>  mkdir mypackages/misc
jo@freya ~>  pkg create mc-nox11
Creating package for mc-nox11-4.8.26
jo@freya ~>  mv mc-nox11-4.8.26.txz mypackages/misc/
jo@freya ~>  pkg repo mypackages/
Creating repository in mypackages/: 100%
Packing files for repository: 100%

Now I can use the "mypackages" directory for a package server.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,626
Messages: 37,981

There's no need for an "ALL" directory, you can use whatever you want.
All is often used because the old package repository structure used it. But yes, it can be any name nowadays. There does need to be a directory:
Code:
     To create a package repository catalogue, specify the top-level directory
     beneath which all the packages are stored as repo-path.  pkg repo will
     search the filesystem beneath repo-path to find all the packages it
     contains.  Directories starting with ‘.’ or named Latest are not
     traversed.
 

Alain De Vos

Daemon

Reaction score: 549
Messages: 1,898

Shouldnt you be able to set up a pkg repository without using openssl or openssl installed ?
I remember using linux and this key thing getting out of hand.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I think that depends on what you have fo signature_type, eg:=

signature_type: "fingerprints",
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

Is it OK to lump together pkgs for different releases of FreeBSD including amd64 and i386 because these errors suggest that pkgs for different releases should be segregated in some way.

pkg: Repository test-repo missing. 'pkg update' required
pkg: wrong architecture: FreeBSD:11:amd64 instead of FreeBSD:13:amd64
pkg: repository test-repo contains packages with wrong ABI: FreeBSD:11:amd64

I have simply lumped together all the pkgs that have been installed over the last couple of years during which time FreeBSD has been updated, but I would still like to service requests from systems that are running older versions.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

To reiterate what I have...

Client

/usr/local/etc/pkg/repos/test-repo.conf:-

Code:
test-repo: {
    url: "file:///mnt/pkg-repo/",
    signature_type: "none",
    enabled: yes
}

Server

The repository is in:-

/var/tmp/pkg

This contains copies of all the *.txz files that are in /var/cache/pkg/ and in addition has packagesite.txz meta.txz meta.conf which were produced when I ran pkg repo.

There are no directories in this directory. Do I need to create an All directory and move all the *.txz files there?

Getting a bit confused by all this...
 

jmos

Active Member

Reaction score: 176
Messages: 243

Is it OK to lump together pkgs for different releases of FreeBSD including amd64 and i386
That can't work. The package system expects them to be in their own repository: amd64 for FreeBSD 11 has to be an own repository as well as i386, and both again for FreeBSD 12 (and 13). F.e. there's a reason why you have to rebuild or upgrade all packages after a major FreeBSD update (and that doesn't mean "install the old packages again", but "install new packages that are not build for the old major version"). Also you cannot have two different versions of one package in one repository.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

So what should the directory structure look like?

Like this?

repo/
i386/
10.0/
11.0/
12.0/
13.0/
amd64/
10.0/
11.0/
12.0/
13.0/

and should pkg-repo() be run in each directory
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,626
Messages: 37,981

To give you an example:
Code:
root@molly:~ # ll /usr/local/poudriere/data/packages/
total 90
drwxr-xr-x  7 root  wheel  15 May 15 15:06 122-release-server/
drwxr-xr-x  7 root  wheel  15 May 15 11:22 13-stable-server/
drwxr-xr-x  7 root  wheel  15 May 15 23:25 130-release-desktop/
drwxr-xr-x  7 root  wheel  15 May 15 14:18 130-release-server/
lrwxr-xr-x  1 root  wheel  18 Nov  7  2020 FreeBSD:12:amd64@ -> 122-release-server
lrwxr-xr-x  1 root  wheel  18 Feb 15 21:24 FreeBSD:13:amd64@ -> 130-release-server
Code:
root@molly:~ # ll /usr/local/poudriere/data/packages/130-release-desktop/
total 118
lrwxr-xr-x  1 root  wheel  18 Feb 11 17:14 .buildname@ -> .latest/.buildname
lrwxr-xr-x  1 root  wheel  20 Feb 11 17:14 .jailversion@ -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel  16 May 15 23:25 .latest@ -> .real_1621113913
drwxr-xr-x  4 root  wheel   9 May 10 00:25 .real_1620599154/
drwxr-xr-x  4 root  wheel   9 May 11 01:21 .real_1620688887/
drwxr-xr-x  4 root  wheel   9 May 12 08:19 .real_1620800359/
drwxr-xr-x  4 root  wheel   9 May 14 16:27 .real_1621002475/
drwxr-xr-x  4 root  wheel   9 May 15 23:25 .real_1621113913/
lrwxr-xr-x  1 root  wheel  11 Feb 11 17:14 All@ -> .latest/All
lrwxr-xr-x  1 root  wheel  14 Feb 11 17:14 Latest@ -> .latest/Latest
lrwxr-xr-x  1 root  wheel  17 Feb 11 17:14 meta.conf@ -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel  16 Feb 11 17:14 meta.txz@ -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel  23 Feb 11 17:14 packagesite.txz@ -> .latest/packagesite.txz
Code:
root@molly:~ # ll /usr/local/poudriere/data/packages/130-release-desktop/All/
total 3626564
-rw-r--r--  1 root  wheel    7539288 May 15 15:22 ImageMagick6-6.9.11.6_3,1.txz
-rw-r--r--  1 root  wheel    2957088 May 15 15:39 RetroArch-1.9.0.txz
-rw-r--r--  5 root  wheel     141928 Apr 16 20:03 aalib-1.4.r5_13.txz
-rw-r--r--  2 root  wheel    9191964 May 14 06:45 adwaita-icon-theme-3.38.0.txz
-rw-r--r--  3 root  wheel     455708 May 12 02:34 alsa-lib-1.2.2.txz
-rw-r--r--  5 root  wheel    3456612 May  5 02:30 aom-3.1.0.txz
-rw-r--r--  2 root  wheel     480564 May 14 01:37 apr-1.7.0.1.6.1_1.txz
-rw-r--r--  5 root  wheel      34496 Apr 16 13:24 argp-standalone-1.3_4.txz
{... and a whole bunch more, I won't bore you with a complete list ..}

I don't have i386 repositories because I have no 32 bit machines and thus no use for 32 bit packages.
 
Top