Pkg package repository using ports-mgmt/poudriere. With or without ZFS.

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,949
Messages: 38,390

set up a dedicated poudriere server with immense disk space and CPU power
You don't need a "massive" server for this. I do all my package building at home on an old Core i5 with 16GB RAM. It just takes a little longer to finish compared to the 24 core, 96GB RAM beast I can use at a client.

which builds tens of thousands of packages that will mostly never be used.
You don't build everything (as in all 40.000+ packages), you feed it a list of things you need.
Code:
     -f file  Build ports listed in the file.

              The path to the file has to be absolute.  Ports must be
              specified in the form of "category/port" and sh(1)-style
              comments are allowed.  Multiple -f file arguments may be
              specified at once.
See poudriere-bulk(8).

I have a poudriere set up for a client, where I need to maintain about 30 FreeBSD servers. Client has no need for desktop or Xorg applications, only Apache, MySQL and an assortment of other web based tools (like Ruby-on-Rails, PHP, etc). I've set this up many years ago and we're still using it right now. The list needs some occasional updates (changes in requirements of the client) but the list is pretty much the same now as it was many years ago. I just let it update its ports tree weekly and run the builds at night. So next morning I have a freshly built and updated package repository. The nice thing about this setup is that I get to decide when to change the default Ruby or PHP version for example, so we can migrate at our own pace. I can also easily revert any updates or temporarily fix ports if needed. I'm also sure all servers have the same versions and settings for everything.
 

Snurg

Daemon

Reaction score: 589
Messages: 1,349

That's awesome!

Question 1:
If I understand correctly, if the package <mynewprogram> is not already in the custom repo, then the process of adding some new port would be (in case the dependencies in our repo are all up-to-date):

1. build the port and its dependencies using poudriere-bulk -f <file_containing_mynewprograms_name>,
2. copy the generated repo files to the repo storage directory of our repo webserver,
3. run pkg-repo to update the package index,

and then the client can just run pkg install <mynewprogram>?

Question 2:
And, when I use pkg install <mynewprogram>, does pkg send some information that I like <mynewprogram> and want my cgi to add it to poudriere's build list, too?
(I ask, because, if possible, I would like to automate the steps described above using my cgi)
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,949
Messages: 38,390

2) don't need to copy it. I have a basic nginx running on the build server itself. You can find configuration examples for nginx and Apache in /usr/local/share/examples/poudriere.
3) No need, poudriere already does this at the end of its run.

nd then the client can just run pkg install <mynewprogram>?
Add the custom repo in /usr/local/etc/pkg/repos/myrepo.conf for example:
Code:
MyRepo: {
  enabled: yes
  url: http://some.internal.host/packages/
}
Also add a /usr/local/etc/pkg/repos/FreeBSD.conf to disable the FreeBSD repositories:
Code:
FreeBSD: {
  enabled: no
}

And, when I use pkg install <mynewprogram>, does pkg send some information that I like <mynewprogram>
pkg(8) just uses fetch(3) to grab the index files (meta.txz and packagesite.txz; these are used to update the locally cached information; /var/db/pkg/repo-*.sqlite) and then fetches the right file directly. There's no CGI or anything like that involved here.

Code:
root@molly:~ # ll /usr/local/poudriere/data/packages/122-release-server/
total 118
lrwxr-xr-x  1 root  wheel  18 Oct 28 19:22 .buildname@ -> .latest/.buildname
lrwxr-xr-x  1 root  wheel  20 Oct 28 19:22 .jailversion@ -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel  16 Jan 25 13:34 .latest@ -> .real_1611578069
drwxr-xr-x  4 root  wheel   9 Jan 18 10:19 .real_1610961573/
drwxr-xr-x  4 root  wheel   9 Jan 20 04:54 .real_1611114841/
drwxr-xr-x  4 root  wheel   9 Jan 22 11:45 .real_1611312342/
drwxr-xr-x  4 root  wheel   9 Jan 22 22:14 .real_1611350051/
drwxr-xr-x  4 root  wheel   9 Jan 25 13:34 .real_1611578069/
lrwxr-xr-x  1 root  wheel  11 Oct 28 19:22 All@ -> .latest/All
lrwxr-xr-x  1 root  wheel  14 Oct 28 19:22 Latest@ -> .latest/Latest
lrwxr-xr-x  1 root  wheel  17 Oct 28 19:22 meta.conf@ -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel  16 Oct 28 19:22 meta.txz@ -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel  23 Oct 28 19:22 packagesite.txz@ -> .latest/packagesite.txz
 

Snurg

Daemon

Reaction score: 589
Messages: 1,349

Damn, is this gosu :)
I want local repos :)
And, I think now I know the most important points I need to start with trying to integrate poudriere into my postinstaller script soon.
Just extract the options information from the package, edit them in a web form, and store in poudriere configuration for later use (install replay, repo cloning,...), let poudriere and after that pkg run :)

Thanks kpa and SirDice :) 👍
 
Top