Pkg package repository using ports-mgmt/poudriere. With or without ZFS.

Once I tried to create a jail. I don't know why, but that last time that I do that I needed to reinstall the system.
I think that was because I disconnected the device that I was using where I had the jail.
Since that, I have panic of even try to create one.
 
The nice thing about poudriere is that you don't need to worry about all that. It's all automated with just a few commands. I would recommend using it with ZFS though. Without ZFS it is incredibly I/O intense when it spawns new build jails.
 
I'm back after some time and I intend to update this HOWTO with the latest changes to the ports system and other stuff. It's likely that I'll change this to default to a ZFS setup but offer alternative settings if you don't want to use ZFS. Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.
 
Updated now to recommend using ZFS as the default configuration and some minor rewrites here and there.
 
I did some heavy handed editing in certain parts. It turned out that you can no longer register an existing ports tree into poudriere and have poudriere adopt that port tree completely. You have to create a new ports tree if you want poudriere to be able to update it as well.
 
kpa said:
Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.
Click to expand...

I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
Has anything changed in that area?
 
spork said:
I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
Has anything changed in that area?
Click to expand...

On those very small systems with 1GBs of RAM it gets a bit tricky to get ZFS working but it can be done with careful tuning and the performance won't be that bad. I would still use UFS as you have done. With 4GBs of RAM however ZFS will work absolutely fine without any tuning and the performance is as good as it gets, you might want to manually limit ARC to about 1GBs out of the 4GBs to guarantee that applications and other parts of the kernel have enough memory available.
 
Only a couple of months old so hope this isn't a necro-bump...;) I have set up a ports-mgmt/poudriere server on a beefy build machine as kpa instructed (thanks!) and it is building a couple of test packages as we speak. In poking around in the directory tree of ports-mgmt/poudriere, I noticed what looks like a copy of the ports tree. This brings me to my first question:

Since ports-mgmt/poudriere is building ports, how to I keep the ports tree updated and which tree do I update?
Also, since I am going to set up this server as the pkg source for my main FreeBSD workstation, what is the easiest way of building /usr/local/etc/ports.txt or must that be done manually?

My assumption is that I will regularly update the ports tree on the build machine and it will be the source of all packages for my normal workstation. Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability.

What is the recommendation for the contents of the build server then, all packages (seems excessive), or how is this situation dealt with normally? I assume the official FreeBSD package repositories have multiple build machines and this is a continual process based on code updates to ports.

Thanks for the HOWTO - was very helpful!
 
Sevendogsbsd said:
Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability.
Click to expand...

poudriere bulk (without the -c option) doesn't rebuild packages if it isn't needed because they're new, updated, have updated dependencies or changed build options. So, just add what you need, and run poudriere bulk on it again.
 
Sevendogsbsd what do you expect? my latest poudriere bulk run is now working on www/chromium for 9 hours and counting.

If you have some heavy-weight ports in your package list, there's (at least) two things you can do to speed up the bulk run:
  • list those fat ports in PRIORITY_BOOST, so they are started as early as possible
  • if you are sure parallel builds are fine, list fat ports in ALLOW_MAKE_JOBS_PACKAGES, so their build uses more than one core. I'm personally leaving this empty because poudriere already builds several ports in parallel, so building a single port in parallel would just increase load -- but if you're in a situation where one single fat port often ends up being the only active job, it might be worthwile to allow parallel building for that one.
 
Just building chromium by configuring and doing a make in the port's directory takes me about an hour and 20 minutes. As an experiment, I tried ports-mgmt/synth and it built devel/llvm60 in about an hour and a half.

I know it takes time, especially with big builds, I just expected it to be faster. No worries, it all completed. Thanks for the pointers!
 
Yes, ports are rebuilt only when there are updates. However, FreeBSD ports doesn't yet have proper tracking if the dependents of a port really require rebuilding. This is results in rather large churn when an important port is updated and possibly over half of your packages are rebuilt even if there was no need for to rebuild anything but the one changed port. For example if there's a small bugfix to security/openssl you can expect a large amount of built packages even if nothing really changed in them.

You have to update and maintain the ports.txt yourself, you're the one who knows which ports you need in your system.
 
Right - got it. I figured out that documenting the ports I needed first was the key. I have a system/process down pat now so it works well. There is definitely a "cascading" of builds after a ports tree update but it's fine: I have a build server so I just fire it up every week or so and refresh the ports tree, then see what needs to be built. It doesn't affect my regular workstation so I can let it run overnight if need be.
 
Do I need to do additional setup before I can use poudriere, setting jail_enable="YES" in /etc/rc.conf for example?

Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?
 
Nyakov said:
Do I need to do additional setup before I can use poudriere, setting jail_enable="YES" in /etc/rc.conf for example?

Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?
Click to expand...

No. Poudriere take care of everything it needs about jail mounting etc.

No. Totally separated configurations.
 
rigoletto@ said:
No. Totally separated configurations.
Click to expand...
Are you sure ? From poudriere(8) :
You can also specify a global make.conf which will be used for all the jails. Any of the following are allowed and will all be used in the order shown:
/usr/local/etc/poudriere.d/make.conf
/usr/local/etc/poudriere.d/<setname>-make.conf
/usr/local/etc/poudriere.d/<tree>-make.conf
/usr/local/etc/poudriere.d/<jailname>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
/usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
Click to expand...
My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.
 
rigoletto@ said:
No. Totally separated configurations.
Click to expand...
So, if jail_name-make.conf present then make.conf ignored?
Why I am asking.
Handbook states: The system make.conf and this new file are combined at build time to create the make.conf used by the build jail

And this howto states that:
Note that the build host's own /etc/make.conf is not used at all because it's assumed to be "tainted"
Click to expand...

So I confused. What "system make.conf" in handbook means?

Question is answered:
hukadan said:
My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.
Click to expand...
 
Nyakov said:
So, if jail_name-make.conf present then make.conf ignored?
Click to expand...
No, they are stacked. Poudriere will always ignore /etc/make.conf.

Code: 
   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf
See poudriere(8).
 
Since your howto is so long, IMHO an index at the top would be appropiate. It helps the reader to build up some mental structure in advance of what is going to come, and with an index you give a reasonable preset.
Missing: 1. Integration of ccache. This is considered common practice. 2. Using svn instead of portsnap to update the portstree, for those who want to stick to the quarterly port branches.
Besides that, the text is structured well and easy to read and understand, which is hard to get right. Thanks.
 
Thank you kpa for your great writeup!

Question:
There are only a number of ports that I need other configurations than the default ones.

I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?

So, if I read correctly, it is possible to configure/use poudriere in this way?
So that it only manages a few selected applications, and I don't need a server that constantly fetches and builds thousands of packages I never use?

Ideally I would like only need to press a button "Update my system!" on my personal web admin tool and let the scripts do the work.

Another issue is, I read portsnap is deprecated and will be removed soon, so mjollnir 's point is very important, as the howto needs to be updated, too...
 
Snurg said:
I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?
Click to expand...
Poudriere just builds everything you tell it to build. This results in a directory full of packages and the required files to make it a repository (pkg-repo(8)). You set up a simple webserver that shares this directory. Then configure your local pkg(8) to fetch its packages from that webserver.

Build everything you need from your own repositories. Mixing your custom settings with the official packages that use default settings is bound to cause dependency issues or other conflicts. Your packages for example might depend on MariaDB 10.4 while the official packages depend on MySQL 5.7. So to prevent this potential nightmare build everything yourself and install only from your own repository.
 
Well this is a good explanation why it's not good to mix "homemade" packages with "official" ones.

The big problem now is, many people don't have the time and resources to set up a dedicated poudriere server with immense disk space and CPU power, which builds tens of thousands of packages that will mostly never be used.

So I think there should then exist a solution about this:

client pkg sends request ->
web server CGI checks whether the package file is already present ->
if it is not present yet, add it to poudrieres list of packages to maintain (for updates),
let poudriere build package (and its dependencies),
update pkg-repo
-> send repo info to client pkg

This would have poudriere to build only what the user actually requests, given pkg is patient enough to wait for the response.
In turn, this would result in a repo small enough to keep locally on the computer.

Does such a solution already exist?
 
You must log in or register to reply here.
Back
Top