Pkg package repository using ports-mgmt/poudriere. With or without ZFS.

B

BSDAppentic3

Guest


Once I tried to create a jail. I don't know why, but that last time that I do that I needed to reinstall the system.
I think that was because I disconnected the device that I was using where I had the jail.
Since that, I have panic of even try to create one.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,309
Messages: 38,829

The nice thing about poudriere is that you don't need to worry about all that. It's all automated with just a few commands. I would recommend using it with ZFS though. Without ZFS it is incredibly I/O intense when it spawns new build jails.
 
OP
K

kpa

Beastie's Twin

Reaction score: 1,846
Messages: 6,318

I'm back after some time and I intend to update this HOWTO with the latest changes to the ports system and other stuff. It's likely that I'll change this to default to a ZFS setup but offer alternative settings if you don't want to use ZFS. Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.
 
OP
K

kpa

Beastie's Twin

Reaction score: 1,846
Messages: 6,318

Updated now to recommend using ZFS as the default configuration and some minor rewrites here and there.
 
OP
K

kpa

Beastie's Twin

Reaction score: 1,846
Messages: 6,318

I did some heavy handed editing in certain parts. It turned out that you can no longer register an existing ports tree into poudriere and have poudriere adopt that port tree completely. You have to create a new ports tree if you want poudriere to be able to update it as well.
 

spork

Active Member

Reaction score: 13
Messages: 159

Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.

I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
Has anything changed in that area?
 
OP
K

kpa

Beastie's Twin

Reaction score: 1,846
Messages: 6,318

I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
Has anything changed in that area?

On those very small systems with 1GBs of RAM it gets a bit tricky to get ZFS working but it can be done with careful tuning and the performance won't be that bad. I would still use UFS as you have done. With 4GBs of RAM however ZFS will work absolutely fine without any tuning and the performance is as good as it gets, you might want to manually limit ARC to about 1GBs out of the 4GBs to guarantee that applications and other parts of the kernel have enough memory available.
 

Sevendogsbsd

Daemon

Reaction score: 679
Messages: 1,125

Only a couple of months old so hope this isn't a necro-bump...;) I have set up a ports-mgmt/poudriere server on a beefy build machine as kpa instructed (thanks!) and it is building a couple of test packages as we speak. In poking around in the directory tree of ports-mgmt/poudriere, I noticed what looks like a copy of the ports tree. This brings me to my first question:

Since ports-mgmt/poudriere is building ports, how to I keep the ports tree updated and which tree do I update?
Also, since I am going to set up this server as the pkg source for my main FreeBSD workstation, what is the easiest way of building /usr/local/etc/ports.txt or must that be done manually?

My assumption is that I will regularly update the ports tree on the build machine and it will be the source of all packages for my normal workstation. Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability.

What is the recommendation for the contents of the build server then, all packages (seems excessive), or how is this situation dealt with normally? I assume the official FreeBSD package repositories have multiple build machines and this is a continual process based on code updates to ports.

Thanks for the HOWTO - was very helpful!
 

Zirias

Son of Beastie

Reaction score: 1,531
Messages: 2,654

Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability.

poudriere bulk (without the -c option) doesn't rebuild packages if it isn't needed because they're new, updated, have updated dependencies or changed build options. So, just add what you need, and run poudriere bulk on it again.
 

Sevendogsbsd

Daemon

Reaction score: 679
Messages: 1,125

Oh that's awesome - even with 12 Xeon cores and 96gb of ram, this devel/llvm60 build is on 2 and a half hours so this is going to get old really fast...
 

Zirias

Son of Beastie

Reaction score: 1,531
Messages: 2,654

Sevendogsbsd what do you expect? my latest poudriere bulk run is now working on www/chromium for 9 hours and counting.

If you have some heavy-weight ports in your package list, there's (at least) two things you can do to speed up the bulk run:
  • list those fat ports in PRIORITY_BOOST, so they are started as early as possible
  • if you are sure parallel builds are fine, list fat ports in ALLOW_MAKE_JOBS_PACKAGES, so their build uses more than one core. I'm personally leaving this empty because poudriere already builds several ports in parallel, so building a single port in parallel would just increase load -- but if you're in a situation where one single fat port often ends up being the only active job, it might be worthwile to allow parallel building for that one.
 

Sevendogsbsd

Daemon

Reaction score: 679
Messages: 1,125

Just building chromium by configuring and doing a make in the port's directory takes me about an hour and 20 minutes. As an experiment, I tried ports-mgmt/synth and it built devel/llvm60 in about an hour and a half.

I know it takes time, especially with big builds, I just expected it to be faster. No worries, it all completed. Thanks for the pointers!
 
OP
K

kpa

Beastie's Twin

Reaction score: 1,846
Messages: 6,318

Yes, ports are rebuilt only when there are updates. However, FreeBSD ports doesn't yet have proper tracking if the dependents of a port really require rebuilding. This is results in rather large churn when an important port is updated and possibly over half of your packages are rebuilt even if there was no need for to rebuild anything but the one changed port. For example if there's a small bugfix to security/openssl you can expect a large amount of built packages even if nothing really changed in them.

You have to update and maintain the ports.txt yourself, you're the one who knows which ports you need in your system.
 

Sevendogsbsd

Daemon

Reaction score: 679
Messages: 1,125

Right - got it. I figured out that documenting the ports I needed first was the key. I have a system/process down pat now so it works well. There is definitely a "cascading" of builds after a ports tree update but it's fine: I have a build server so I just fire it up every week or so and refresh the ports tree, then see what needs to be built. It doesn't affect my regular workstation so I can let it run overnight if need be.
 

Nyakov

Member

Reaction score: 9
Messages: 49

Do I need to do additional setup before I can use poudriere, setting jail_enable="YES" in /etc/rc.conf for example?

Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?
 

rigoletto@

Daemon
Developer

Reaction score: 1,251
Messages: 2,291

Do I need to do additional setup before I can use poudriere, setting jail_enable="YES" in /etc/rc.conf for example?

Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?

No. Poudriere take care of everything it needs about jail mounting etc.

No. Totally separated configurations.
 
H

hukadan

Guest


No. Totally separated configurations.
Are you sure ? From poudriere(8) :
You can also specify a global make.conf which will be used for all the jails. Any of the following are allowed and will all be used in the order shown:
/usr/local/etc/poudriere.d/make.conf
/usr/local/etc/poudriere.d/<setname>-make.conf
/usr/local/etc/poudriere.d/<tree>-make.conf
/usr/local/etc/poudriere.d/<jailname>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.
 

Nyakov

Member

Reaction score: 9
Messages: 49

No. Totally separated configurations.
So, if jail_name-make.conf present then make.conf ignored?
Why I am asking.
Handbook states: The system make.conf and this new file are combined at build time to create the make.conf used by the build jail

And this howto states that:
Note that the build host's own /etc/make.conf is not used at all because it's assumed to be "tainted"

So I confused. What "system make.conf" in handbook means?

Question is answered:
My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,309
Messages: 38,829

So, if jail_name-make.conf present then make.conf ignored?
No, they are stacked. Poudriere will always ignore /etc/make.conf.

Code:
   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf
See poudriere(8).
 

Mjölnir

Daemon

Reaction score: 1,503
Messages: 2,114

Since your howto is so long, IMHO an index at the top would be appropiate. It helps the reader to build up some mental structure in advance of what is going to come, and with an index you give a reasonable preset.
Missing: 1. Integration of ccache. This is considered common practice. 2. Using svn instead of portsnap to update the portstree, for those who want to stick to the quarterly port branches.
Besides that, the text is structured well and easy to read and understand, which is hard to get right. Thanks.
 

Snurg

Daemon

Reaction score: 592
Messages: 1,349

Thank you kpa for your great writeup!

Question:
There are only a number of ports that I need other configurations than the default ones.

I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?

So, if I read correctly, it is possible to configure/use poudriere in this way?
So that it only manages a few selected applications, and I don't need a server that constantly fetches and builds thousands of packages I never use?

Ideally I would like only need to press a button "Update my system!" on my personal web admin tool and let the scripts do the work.

Another issue is, I read portsnap is deprecated and will be removed soon, so mjollnir 's point is very important, as the howto needs to be updated, too...
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,309
Messages: 38,829

I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?
Poudriere just builds everything you tell it to build. This results in a directory full of packages and the required files to make it a repository (pkg-repo(8)). You set up a simple webserver that shares this directory. Then configure your local pkg(8) to fetch its packages from that webserver.

Build everything you need from your own repositories. Mixing your custom settings with the official packages that use default settings is bound to cause dependency issues or other conflicts. Your packages for example might depend on MariaDB 10.4 while the official packages depend on MySQL 5.7. So to prevent this potential nightmare build everything yourself and install only from your own repository.
 

Snurg

Daemon

Reaction score: 592
Messages: 1,349

Well this is a good explanation why it's not good to mix "homemade" packages with "official" ones.

The big problem now is, many people don't have the time and resources to set up a dedicated poudriere server with immense disk space and CPU power, which builds tens of thousands of packages that will mostly never be used.

So I think there should then exist a solution about this:

client pkg sends request ->
web server CGI checks whether the package file is already present ->
if it is not present yet, add it to poudrieres list of packages to maintain (for updates),
let poudriere build package (and its dependencies),
update pkg-repo
-> send repo info to client pkg


This would have poudriere to build only what the user actually requests, given pkg is patient enough to wait for the response.
In turn, this would result in a repo small enough to keep locally on the computer.

Does such a solution already exist?
 
Top