PF How to allow Public Key Auth logins while blocking IP addresses??

Hi, I couldn't seem to find an answer with several searches so here goes...
I am running FreeBSD 12.3-RELEASE-p5 GENERIC. I have PF configured and working well for my needs. It blocks all SSH logins except for specific IP addresses. So, I can only log in to my server from 2 offices, my home, and a backup location.

But if I'm on vacation and I need to log into the server, I'm blocked. I was thinking I could set up Public Key Auth on my laptop but the random vacation IP would still be blocked by PF.

Is there a rule I could make in PF that would allow Public Key logins before it gets to the rule that blocks IPs?

I don’t believe so. pf is filtering at TCP level, whereas public key exchange happens after the port 22 connection has been established.
Thanks! And yeah, that makes sense. I already have my server configured to use another port besides 22. That significantly reduced the number of bots trying to log in. I still like using an IP whitelist as an extra layer though.

Maybe I could pick a local internet provider in vacation area, and temporarily whitelist their whole IP block?

Is having an IP whitelist overkill?
Another option - but more work and $$$s - is to have a machine (let's call it X) at another location (with a static IP that you add to your whitelist so that you can access your server) that you (temporarily) allow connections from any IP, and then you ssh into that machine and setup rules to tunnel/relay your traffic to the server.

So before you go away, log into this machine X, allow other IPs to log-in to it. Use ssh with no-passwords, no-root-login, AllowUsers set - lock it down as much as possible. Check the machine can connect to your server. At this point no relay/tunneling set-up.

You're on holiday. You need to connect to the server. You log into machine X from your holiday IP (you've allowed any IP to connect just while you are away). You set up any tunnel/relay and then your holiday machine can access your server via machine X. You finish the connection, log back into X, tear down the relay/tunnel. But X will still be available to set it up again whilst you're on holiday.

When you get home, log back into X and disable "IPs from anywhere" until you go on holiday again.
Change the default SSH port to reduce the log noise from the scan bots then disable the PasswordAuthentication and PAM auth (KbdInteractiveAuthentication no or the old deprecated variable ChallengeResponseAuthentication no) and use PKI as only auth metod.
Also to protect your private key add a password to it.
FTR: Internet-accessible sshd should *always* only allow key-based logins.

Instead of blocking everything, just use security/sshguard and maybe set up pf-badhost [1] and completely drop *any* traffic from hosts in those tables, not only for ssh. This removes _a lot_ of noise on all services and these PF tables can be distributed between hosts.