Building isn't necessary -- the FreeBSD packages are available as binaries.
The only caveat is that they aren't signed packages, so that being a dealbreaker is the only benefit of building yourself that is gained.
I do need to get around to signing these things.
I build PHP specifically with certain options (ZEND_THREAD_SAFTEY, MAILHEADER) and also other ports with some non-default options.
The number of ports I build with special options has decreased over time, though. But some will always stay "special".