The idea behind tweaking the make.conf is to customize your builds for your CPU and your needs (CFLAGS, etc). This is why you rebuild the world after install, to exclude redundant code and make most of your CPU.
Next step is to customize the software you are installing by enabling options and features you need and excluding what you don't need, to reduce the size of install.
If all this does not make any sense to you then binary install will do.
I doubt you will notice any slowdown, unless your hardware is really old.