Package management at scale

AngusBuchanan

AngusBuchanan

Hello,

I have a large number of machines (several hundred) that I am considering migrating from Ubuntu to FreeBSD. The one area where I need to decide on my approach is the area of package management.

I've read previous threads in this area of the forum, and the FreeBSD handbook, and it seems to me that the recommended approach to managing software is to use ports rather than packages. My concern is that if I need to roll out changes over a large number of machines, packages would be much much faster. Obviously there are not packages for everything I need, so I was considering building packages from ports on a build server, and serving them as packages, and installing them with pkg_add -r - so not using any of the precompiled versions - just whatever I built on the build server, tracking -STABLE. I was thinking I could keep on top of this process with portaudit and portmaster on the build server.

My fear is that the package management tools aren't really very clever and don't seem to handle reciprocal depencencies or even upgrades. It also seems to be the advice that packages shouldn't be upgraded in a piecemeal fashion, as this leads to dependency problems.

I'm wondering, then, if the best approach is actually to just rebuild the whole estate fairly regularly? Or upgrade the whole lot every so often.

This isn't ever so different from Solaris/SunOS patch clusters - wait for a while, then upgrade the whole lot all at once. It's just that with several hundred machines, this is a big job... and one which needs some orchestration to not be alarmingly manual.

Sorry if this is a little rambling - I'm still trying to get my thoughts clear on this. Am I on the right track, or have I missed a trick?
 
Building your own packages is the best idea in my opinion. After that you can simply export /usr/ports/packages/ or some other place where you've stored them. To 'circumvent' all the dependency interaction I do all my building inside a clean jail. That way I'm sure everything is build correctly and in order. For my other machines I simply pkg_delete(1) everything and install everything I need. Again, no dependency problems because all those packages came from the same build run. It takes me about 20 minutes to 'refresh' a machine this way.

As for when to do that, that depends. I usually keep an eye on the ports tree, depending on what got updated I'll start off a new build. Keeping track with portsaudit is probably the best way and only update when there are security issues.
 
What we do is similar to what SirDice suggests. We have a build machine with a clean jail to build the packages for each port we use on our farm, so that we can use non-default OPTIONS etc. We use ports-mgmt/portupgrade and its portinstall -p to build the packages. Then, we export the whole /usr/ports to each node (obviously not to all of them at the same time, but this step can be easily scripted I guess) and use portinstall -PP to install new packages or portupgrade -PP to use the pre-built packages when upgrading them. I guess it is also possible to use ports-mgmt/portmaster (which seems to be much faster than portupgrade) for all of these tasks.

This approach is quite easy when you have machines with pretty much the same configuration, however it's getting more complex when you have mixed archs (amd64/i386 etc) or need different OPTIONS for the same package on some of the machines.

In such case I'd recommed you to check ports-mgmt/tinderbox - http://tinderbox.marcuscom.com/ which might be of help to resolve above mentioned issues.

In comparision to what SirDice describes this approach does not require uninstalling all of the installed apps and rolling them back at the same time which seems to be an overkill to me.

Should you have any additional questions, feel free to ask.
 
danger@ said:
In comparision to what SirDice describes this approach does not require uninstalling all of the installed apps and rolling them back at the same time which seems to be an overkill to me.
Click to expand...
Yeah, there are probably smarter ways of doing it but this works for me. And I can be sure all dependencies are correct.
 
danger@ said:
What we do is similar to what SirDice suggests. We have a build machine with a clean jail to build the packages
Click to expand...

Just to clarify - does this simply mean you fire up a brand new jail, and do the building, then throw it away?

for each port we use on our farm
Click to expand...

What about stuff in base?

we export the whole /usr/ports to each node
Click to expand...

What's the recommended way to do this? NFS? Or do you recommend making packages available over http/ftp?
 
AngusBuchanan said:
Just to clarify - does this simply mean you fire up a brand new jail, and do the building, then throw it away?
Click to expand...
Yep, that's basically it. Put the jail on it's own filesystem. A newfs is much, much quicker then rm -rf ;)

What about stuff in base?
Click to expand...
Export /usr/src/ and /usr/obj/. You can mount those on the destination systems. Each system can have it's own /etc/src.conf in case you want to skip installing bits. Different kernels can be set using KERNCONF in /etc/make.conf. The actual building of all the kernels can be done on the build server.

What's the recommended way to do this? NFS?
Click to expand...
That's how I do it. Simplest way to set up. You can export /usr/src/, /usr/obj/ and /usr/ports/ read-only if you're worried about security. If you do happen to need to build something on one of the other machines you can set WRKDIRPREFIX to a writable (local) filesystem.

Or do you recommend making packages available over http/ftp?
Click to expand...
That's another way to do it. You can point PACKAGESITE to your own 'distribution' server.
 
SirDice said:
Yep, that's basically it. Put the jail on it's own filesystem. A newfs is much, much quicker then rm -rf ;)
Click to expand...

In our case, we keep the jail around - we use it later to do the package upgrades. The jail is being used just because we do not want to blow the host system with all the packages (which they obviosly need to be kept installed) and as was mentioned it is easy to throw it away when it isn't needed anymore. Note that the jail actually doesn't need to be really running - you can just install the "jail" (distribution), mount devfs, chroot inside the tree and start the building process.

As of the other questions, SirDice provided good answers :)
 
danger@ said:
In our case, we keep the jail around - we use it later to do the package upgrades.
Click to expand...
That's why I remove it. If you do the 'standard' update you still need to keep an eye on /usr/ports/UPDATING and keep track of the correct update order. By starting off 'fresh' I don't need to worry about that.
 
BTW, I believe people here would be interested whether you guys have dicided about the migration, and what was its result in the end, or if you have decided to stay with Ubuntu - what was the reasons etc.

Hopefully we were able to answer your questions in enough detail.
 
Hello!

We've decided to move some small sites over, and to deploy some customers onto FreeBSD in preference to Ubuntu. If this pilot goes well, we can look at moving everything else. We'll also start deploying basic infrastructure on FreeBSD in preference to Linux. I'm happy with package management now, thank you.

I'll keep you posted!
 
SirDice said:
Building your own packages is the best idea in my opinion. After that you can simply export /usr/ports/packages/ or some other place where you've stored them. To 'circumvent' all the dependency interaction I do all my building inside a clean jail. That way I'm sure everything is build correctly and in order. For my other machines I simply pkg_delete(1) everything and install everything I need. Again, no dependency problems because all those packages came from the same build run. It takes me about 20 minutes to 'refresh' a machine this way.

As for when to do that, that depends. I usually keep an eye on the ports tree, depending on what got updated I'll start off a new build. Keeping track with portsaudit is probably the best way and only update when there are security issues.
Click to expand...

Hi SirDice,

Please could you go into more depth about this process? Or link if it's already documented?

It made sense to me at first glance, but if you rebuild a package from scratch in a clean environment and move it over to your system, surely all the dependencies will still be missing? Unless you have either statically linked every lib it needs so it's one massive standalone bin (is this even possible?). Or you replace the dependencies too.... but that would then be no different from just deinstalling and reinstalling in the system to begin with? Can't see how you're avoiding dependency hell here?

I have a fresh install (for the x'th time) after having broken something or another by installing a package when ports didn't build and trying the ports out when the packages didn't want to work etc etc... they never seem to want to mix, but I prefer the ports route so not too fussed.

FreeBSD 8.2-RELEASE, GENERIC, i386.
 
You must log in or register to reply here.
Back
Top