Security is always a compromise. With an infinitely large team of administrators and infinite time, you can make a system perfectly secure. Consider the price you'll pay for that though: Not only the salaries of all those people, and the long wait times while they work on securing things. But also their constant intrusiveness (security is usually done at the cost of convenience), and the downtime (like all systems, security isn't perfectly reliable).
So the art of doing security is (like all of engineering) is the art of compromise: What can I do that has low cost, low complexity, and is pretty good? How many of these things can I layer together? Of all the possible approaches, which has the best cost/benefit? Given that no approach is ever 100% reliable and safe, how many layers do I need to implement, until I feel comfortable?
In that spirit: Editing /etc/ssh/sshd.conf to turn off unwanted sources of ssh connections is relatively easy. The cost is that one has to sit down (for a minute or an hour, depending on experience) to read and understand the
sshd_config(5) man pages. Decent security for low cost. Problem is: Now there are explicit addresses in that config file, and if the network is reconfigured, things will go pear-shaped. The opportunity is that one should also think through many of the other options in that config file (for example: Should root login be allowed? Should passwords or keys or both or neither be allowed?), and think about ssh security holistically.
On the other side, editing /etc/pf.conf is also an excellent idea. But the effective way to do it is (as gpw928 said above) is not to open or close on port, but to think about the host's whole security posture: what services does it need to provide? Which of those services are actually really worth the cost of making them secure enough? Who is likely to be attacking, and what might they be after? Is it even worth it to secure certain things, while other things need to be (unfortunately by their nature) wide open? For example, for a machine that is strictly on an internal network that is secure, blocking things by default is just creating a hairball for no benefit. And consider that pf.conf also depends on network configuration (it tends to have both interface names and address ranges explicitly written in it).
And finally, consider other layers. For example, I really like the proposal that gpw928 also mentioned: Move ssh to a different port. It is not secure against determined attackers, but it means that the dumb script kiddies that only probe port 22 have no chance.
), never heard of it before I really hope the dev won't let it die. 