Parallel build on multiple computers

There are parallel build systems. Of the ones compatible with the normal Makefile syntax, Pmake comes to mind (although I don't know where to get it). Of the more modern ones, Bazel can do it. However, the problem with parallel build is that it tends to overstress things. To begin with, you need to have your dependencies exactly correct in the Makefiles or logical equivalent; doing multiple compiles in parallel really exposes mistakes in those. Also, a parallel build typically uses a distributed file system underneath. Doing this with NFSv2/3 is problematic, since traditional make relies on the mtime attribute of files, which NFS implementations handle in a very sloppy fashion, leading to make errors. I have done it with modern cluster file systems, and at least there it works. But my experience has been that the IO demands of parallel building tend to get limited by the file system infrastructure, in particular networking. For example, if you are just using gigabit Ethernet and one central file server, you are probably better off running make on multiple cores of a single machine, you will get better throughput.

Again, the answer whether multi-node make helps or hurts performance really depends on (a) the underlying hardware, and (b) the build system and the target.
 
Back
Top