pass out log all:
Why log all packets?
Where in your ruleset is this rule. Placement where it is in the ruleset matters, hugely. Also, quick rules are checked before this rule.
My VPN, using ssh, has an interface. Do you use IPsec? IPsec, designed with IPv6 in mind, has no concept of...