relayd with a source IP NAT?

I'm attempting to use relayd to act as an extremely simple load-balancer between two machines, however all 3 machines are on entirely different public networks.

However, it seems that relayd is just creating a rdr rule in pf, not a nat rule as well, so the packet is redirected but maintains the original source IP. This means of course that the backend machine sees the SYN packet come in with the original client IP, and so it sends the SYN-ACK packet back to the original client. Of course, the original client sent the SYN to the IP of the load balancer, and so everything fails.

table <primary> { }

table <secondary> { }

redirect imaps {
    listen on port 993 interface xn0
    forward to <primary> check ssl
    forward to <secondary> check ssl

nat-anchor "relayd/*"
rdr-anchor "relayd/*"
anchor "relayd/*"

(I added the "nat-anchor" line in hopes that would do what I needed, but as far as I can tell there was no effect.)

I'm looking to see if there is a way to make relayd create a NAT rule as well so that everything will work properly, or if I need to switch to another solution. (If I do need to try another solution, any suggestions will be greatly appreciated.)