I too have decreased the defaults from 6 builders x 4 jobs, to 4x3. I've also tried 3x4, but I am still undecided.
Assuming your system has 4 cores, try with 3x4 or even 3x3, with 8 GB RAM + 4 GB swap you may still need a bit of luck to not run out of that resources, where two buiders will be probably the "safe" setting.
Synth, by using tmpfs, make large use of the RAM, and that's good, the drawback is that when the RAM is limited there is no room for 'disk cache', and that is bad, and even worse when forced to make extensive use of swap, the extreme case being running out of swap completely.
Fact is that many (most) packages will require little RAM (that is on average 1.25 GB x builder, as desumed from Synth man page), but some will require more RAM, in those case the system will fallback to use disk I/O. (tmpfs filesystems are backed from swap).
Synth build up a clean environment for each packages to be build, (local base), and to do so must read the base files from disk and write them to tmpfs, I would expect that very often the base files are the same for each port and fetching them from the disk cache would be much faster than reading from disk, similar reasoning for build dependencies, that's why in my opinion could be better to keep down the number of builders and possibly allow for more disk cache.
At the end, the whole job might be faster by using less builders.