I had it on 10.4 so, that should not be the problem, also RAM usage should not be the problem, I also was building that on a machine with 32 GB RAM, that's all more than sufficient.
I would rather look into the toolchain. At the moment I am trying to build world first and then the kernel, to see if that improves something.