Solved Table with exclusion for an IPv6 range

I have a persistent table to ban IPs which ask too much to my server, but I want to preserve my local IPs from banning.
For IPv4, something like
in the table seems to work.

But for IPv6, something like
doesn't prevent a ban for 2a02:40e0:b200:400:604b:f990:77b6:110a. I don't understand why. Is the "!" syntax different for IPv6?
Tables are supposed to always use the most narrowly matching entry. So if you have, for example, a table and ruleset like so:
table <badguys> { !2a02:40e0:b200:400::/64 }
block from <badguys>
block to <badguys>
...and you later dynamically add individual hosts (i.e. a /128 entry) to the badguys table, they will override your inverse mask as /128 is much more specific than /64.

Not sure if this is your problem as I'd have to see the ruleset to know what you are doing.
Thank you for your answer. I think this is the problem, but do you know a way to preserve a range of IPs from banning?
There are several ways to do so. The naive way is to simply use a quick pass rule for the whitelisted table, and though this works, it won't let you do any additional filtering (and if you're like me, you probably still want to block certain ports for the whitelisted users).

A better way would probably be policy based filtering as it lets your other rules still apply, and only whitelists those that would otherwise be in the badguys table, e.g., something like this:
table <badguys> { }
table <goodguys> { 2a02:40e0:b200:400::/64 }

# can have rules here ...

block from <badguys> tag CHECKWLF
block to <badguys> tag CHECKWLT

# ... and other rules here ...

pass from <goodguys> tagged CHECKWLF # needs to be after the badguys table checks above
pass to <goodguys> tagged CHECKWLT # ditto above

# ... and still more rules here if you want
This way your whitelist table can override the blocked entries in the <badguys> table, no matter how specific they were. The down side is it has to check several rules and tables, so there might be some performance impact. There may be a better way to do this as well, as I've never actually attempted this in any of my rulesets.
Thank you very much for this explanation!
If I well understood, as my block rule is a quick one, if I don't change it, I need to put another quick rule to pass whitelist before:
pass quick proto { tcp, udp } from <goodguys>
block quick proto { tcp, udp } from <badguys>
But with the disadvantages you explained. I will study a solution with tagging, but as my rules are already very long (300 lines with a dozen of jails), it could be hard to manage.