pkg repo

OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

Wow!

I'm trying figure out exactly how to make a mini version of this..

Presumably your /usr/local/etc/pkg/repos/my-repo.conf contains a URL pointing at /usr/local/poudriere/data/packages/ where you have the directories you mentioned such as /130-release-desktop/ and in that you have /All/ where you put your pkgs. So do you run pkg-repo ()on every subdirectory? And when a client runs pkg install how does the server know which directory to get its pkgs from? Is that via /usr/local/etc/pkg/repos/my-repo.conf on the client?

When you normally run pkg install, FreeBSD pkg server decides which version to give you without you specify the ABI, although that is probably done behind the scenes and does not need to be done explicitly..

Apologies for so many questions. Enquiring minds need to know :)
 

Alain De Vos

Daemon

Reaction score: 548
Messages: 1,898

When you run "pkg install" it goes first looking to all files in /etc/pkg and then in /usr/local/etc/pkg/
The contents of these files contain the info on where and how to find the packages.
A line like "url: "file:///poudriere/data/packages/poujail-pouports" says it is on this pc with this path.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

Looking at /etc/pkg/FreeBSD.conf it says:-

# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
# mkdir -p /usr/local/etc/pkg/repos
# echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
So it must first look to see if such a file as /usr/local/etc/pkg/repos/FreeBSD.conf exits

There is also /usr/local/etc/pkg.conf which may or may not exist.

Somewhere along the line an ABI is passed to the pkg server which must direct it to the corresponding directory to service this ABI. Don't know how or if the repository should be structured to cope with the ABI.
 

Alain De Vos

Daemon

Reaction score: 548
Messages: 1,898

There is a variable ${ABI}. I don't know where this variable is defined. But it is a good question.
To print your ABI:
Code:
pkg -vv | grep ABI
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,624
Messages: 37,960

Presumably your /usr/local/etc/pkg/repos/my-repo.conf contains a URL pointing at /usr/local/poudriere/data/packages/ where you have the directories you mentioned such as /130-release-desktop/ and in that you have /All/ where you put your pkgs. So do you run pkg-repo ()on every subdirectory? And when a client runs pkg install how does the server know which directory to get its pkgs from? Is that via /usr/local/etc/pkg/repos/my-repo.conf on the client?
As you can tell by the directory this is what poudriere produces. But that's not relevant in this case. I have that /usr/local/poudriere/data/packages/ directory shared with a webserver as http://ports.myserver/packages. Each of those sub directories is a different repository. So on my 13.0 desktops I point my-repo.conf to http://ports.myserver/packages/130-server-desktop/, my 12.2 servers point to http://ports.myserver/packages/122-release-server/, etc.

There is a variable ${ABI}. I don't know where this variable is defined. But it is a good question.
It's generated in the pkg(8) code itself. It's based on the information of the machine it's running on. So on a 13.0-RELEASE AMD64 you get FreeBSD:13:amd64, on a 12.2-RELEASE i386 you get FreeBSD:12:i386, etc. If you use that ${ABI} variable in the repository URL it will automatically get translated by pkg(8). That makes it easy to have the "same" URL for different versions and architectures in the config. I've added those symlinks in my http://ports.myserver/package directory so I can use it for my own repositories too and have it "automagically" switch to the correct repository for that version after I upgraded a machine from 12.2 to 13.0 for example. If I had used the "hardcoded" http://ports.myserver/packages/122-release-server/ I would have needed to modify it or else it's going to try to install 12.2 packages on a 13.0 machine after I upgraded it.
 

Alain De Vos

Daemon

Reaction score: 548
Messages: 1,898

Indeed. By using this ABI variable you can easily set up your own different repositories while having the same config file on the different machines.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

Another attempt to get this working... I now have a bare-bones system, plain FreeBSD (13.0) no pkgs and would like to see if I can get anywhere with my pkg repo.

Now that I think I've set it up, I understand I need to run pkg -vv to see if the repo is recognised... but... after running that I get:-

The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from file:///mnt/pkg-repo, please wait...
pkg: Error fetching file:///mnt/pkg-repo/Latest/pkg/txz: No such file or directory
A pre-built version of pkg could not be found on your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.

Now if I have already used pkg somewhere, can I expect to find a copy of pkg.txz somewhere on my system, ie in /var/cache/pkg for example?
 

jmos

Active Member

Reaction score: 176
Messages: 243

Now if I have already used pkg somewhere, can I expect to find a copy of pkg.txz somewhere on my system, ie in /var/cache/pkg for example?
Depends. When you're using packages there's a package file by default, but not named as "pkg.txz":
Code:
root@freya ~>  pkg clean -a
[…]
root@freya ~>  pkg install -f pkg
[…]
root@freya ~>  ls /var/cache/pkg/
total 8576
lrwxr-xr-x  1 root  wheel    25B 29.05.2021 17:55 pkg-1.16.3.txz -> pkg-1.16.3~035fc218ad.txz
-rw-r--r--  1 root  wheel   8.3M 01.01.1970 01:00 pkg-1.16.3~035fc218ad.txz
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I haven't yet fathomed how pkg install pkg manages to retrieve pkg-1.16.3.txz, but I believe that packagsite.yaml provides some sort of index for want you want and what is available.

As far as building a local pkg repository I think I maybe need to have separate repos for FreeBSD versions 11, 12 and 13 as well as corresponding ones for i386 and amd64. Originally I thought I could lump them all into one repository but that doesn't seem to work.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I finally managed to make some progress, at least I've managed to install a couple of pkgs. These are not pkgs built by me but copied from various /var/cache/pkg/ directories.

In one instance I'm trying to install misc/pdmenu. One of the dependencies is png: 1.6.37 and I do have a png-1.6.37~c6ad91dc43.txz in my repo, and there is also a linked file in /var/cache/pkg.

Is there a way to create these linked files in the repo? Does pkg-repo()do that?

I'm coming back to a directory I created over a year ago before getting fully familiarised with pkg repositories.
 
OP
B

balanga

Son of Beastie

Reaction score: 217
Messages: 4,024

I think I have managed to create the backend of my pkg-repo now, ie I can run pkg install -y xyz and xyz will be installled, but I only have a couple of pkgs in my repo at the moment. What I would like to do is populate it with the contents of various /var/cache/pkg/ directories on various systems I have, but not sure how straightforward a process that is. Can I just copy the contents of those directories to my pkg-repo and run pkg-repo() or is there any way to check the ABI of pkgs in cache. At the moment my repo is setup as FreeBSD:12:amd64.
 

Jose

Daemon

Reaction score: 866
Messages: 1,052

The man page for pkg-repository(5) explains the layout in detail. Poudriere places all the packages it builds in the correct filesystem organization and creates the metadata files for you. All you have to do is serve up the the Poudriere directory using Nginx or similar.
 

jmos

Active Member

Reaction score: 176
Messages: 243

What I would like to do is populate it with the contents of various /var/cache/pkg/ directories on various systems I have, but not sure how straightforward a process that is. Can I just copy the contents of those directories to my pkg-repo and run pkg-repo() or is there any way to check the ABI of pkgs in cache.
Of course you can copy & use the package files from there (the symlink name is the one you need in your package repository), but: Why? Again: You can simply create all of them in one go (instead of relying on a maybe incomplete cache). Also you won't get a lolly if you still want to mix ABIs into the same repository (you already know that the pkg command complains about that).
 
Top