Bear in mind that you will run into situations where it's very difficult to meet the goal you seek here. E.g. if you are building port A with the system compiler, port B with a compiler from ports, and they have a shared library dependency in common.
Personally, I recommend sticking to the default system compiler unless you have a really good reason for changing it or it is overridden by the ports tree for a particular port. The most widely tested scenario is the one where you do not locally add any special overrides to default behaviour. You lose supportability by going your own way, as most people using the port will not have your abnormal build configuration. It invites random future failures when updating the ports tree, as it will not have been tested for your setup.
Clang is pretty good these days, so there really should not be a good reason for using something else if you are on a recent release of the OS. Of course, there will always be narrow cases where there is a good reason for picking a specific compiler, just in general it shouldn't be necessary and may cause far more pain than any benefits gained from it.