Poudriere with parallel computing...

[JamesElstone]

Hi all,

Quick question, could ports-mgmt/poudriere performance be enhanced by parallel computing over multiple machines rather than just multiple CPUs / cores on a single node?

Something like Poudriere on a BeoWolf cluster (application of net/pvm) or some other distributed computing construct?

Anyone heard of something like this or methods to distribute compilation of ports to aid compile time?

Any thoughts / experience / feedback welcome!

73,
James

 

[Beastie7]

Theoretically it could if someone were to implement batch processing into Poudriere. But that would complicate things and provide little benefit for it's purpose. It's easier and less tedious to scale out (more cores) than to fiddle with the networking side of things with scaling out. For something as simple as building packages, it wouldn't be needed.

A cool idea IMO would be a use a different platform with more parallelism per-core (ie. POWER8) for building packages. But then I don't think you could cross-build x86 packages on a different instruction set architecture.

 

[kpa]

There is the daemon mode that is the batch processing feature but there are no hooks for spreading the work load over network to multiple hosts. I can see the main problem in how to co-ordinate multiple jails over multiple hosts in a coherent manner.