PF Return traffic: af-to vs. rdr-to rules

[janedenone]

I run an IPv6-only webserser in a VNET jail with the following pf configuration:

block all
pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 80 af-to inet6 from $jail_gateway to $webjail_ip6 port 80
pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 443 af-to inet6 from $jail_gateway to $webjail_ip6 port 443
pass in quick on $ext_if inet6 proto tcp from any to ($ext_if) port 80 rdr-to $webjail_ip6 port 80
pass in quick on $ext_if inet6 proto tcp from any to ($ext_if) port 443 rdr-to $webjail_ip6 port 443

While IPv4 requests are handled properly, return traffic for IPv6 requests is blocked until I add the following rule:

pass out quick on $jail_if inet6 proto tcp from any to $webjail_ip6 port {80, 443}

Why does rdr-to need this additional rule, while af-to does not?