Hi,
The short version:
Is it possible to create NAT and redirect rules for connections initialized on the local machine to an IP address on a local CARP interface?
Longer version and background:
For various reasons I need to set[]up a fairly complicated clustered SMTP service. One layer of this cake is a combined load balancer and spamfilter. Once the mail goes through the spam filter they should be relayed to a single CARP:ed IP address (let's call it $carp_ip). This $carp_ip is configured so that any of the machines in this layer can "own" the address.
I set up the following rules in pf.conf to NAT and redirect connections to $carp_ip to the SMTP nodes in the next layer:
I tested by using telnet from one of the server that were not CARP master, which connects just fine. However, when I try to connect with telnet from the CARP master I simply got a "Connection refused".
Figuring that was kind of odd, I fired up my trusted tcpdump and found that telnet did not send the request on $int_if, or even $carp_if, but on the loopback device:
Well, that's odd, so figuring there was just one interface I didn't want this NAT/redirect rules to apply for I simply changed the rules to this:
Now telnet does not respond with "Connection refused", it just times out after a while. tcpdump on lo0 tells me that it sends a request, but receives no reply. But now interesting things happen on $int_if:
So the redirect and NAT rule works as expected, but the reply is never returned to telnet which, for some reason, sends the request on lo0 (even when a specific source address is specified)... Or something?
So could anyone explain what's going on here? And it is even possible to do what I want?
The short version:
Is it possible to create NAT and redirect rules for connections initialized on the local machine to an IP address on a local CARP interface?
Longer version and background:
For various reasons I need to set[]up a fairly complicated clustered SMTP service. One layer of this cake is a combined load balancer and spamfilter. Once the mail goes through the spam filter they should be relayed to a single CARP:ed IP address (let's call it $carp_ip). This $carp_ip is configured so that any of the machines in this layer can "own" the address.
I set up the following rules in pf.conf to NAT and redirect connections to $carp_ip to the SMTP nodes in the next layer:
Code:
rdr on $int_if inet proto tcp from any to $carp_ip port 25 -> <smtp_nodes>
nat on $int_if inet proto tcp from $local_net to <smtp_nodes> port 25 -> $carp_ip
...
pass in quick on $int_if inet proto tcp from any to $carp_ip port 25I tested by using telnet from one of the server that were not CARP master, which connects just fine. However, when I try to connect with telnet from the CARP master I simply got a "Connection refused".
Figuring that was kind of odd, I fired up my trusted tcpdump and found that telnet did not send the request on $int_if, or even $carp_if, but on the loopback device:
Code:
# tcpdump -ni lo0 tcp port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes
16:43:00.581612 IP 10.0.0.2.62291 > 10.0.0.5.25: Flags [S], seq 2093852831, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 25231064 ecr 0], length 0
16:43:00.581649 IP 10.0.0.5.25 > 10.0.0.2.62291: Flags [R.], seq 0, ack 2093852832, win 0, length 0Well, that's odd, so figuring there was just one interface I didn't want this NAT/redirect rules to apply for I simply changed the rules to this:
Code:
rdr on !$ext_if inet proto tcp from any to $carp_ip port 25 -> <smtp_nodes>
nat on !$ext_if inet proto tcp from $local_net to <smtp_nodes> port 25 -> $carp_ip
...
pass in quick on !$ext_if inet proto tcp from any to $carp_ip port 25Now telnet does not respond with "Connection refused", it just times out after a while. tcpdump on lo0 tells me that it sends a request, but receives no reply. But now interesting things happen on $int_if:
Code:
# tcpdump -ni $int_if tcp port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on $int_if, link-type EN10MB (Ethernet), capture size 65535 bytes
16:50:20.503866 IP 10.0.0.5.56150 > 10.0.0.73.25: Flags [S], seq 3129724327, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 25670986 ecr 0], length 0
16:50:20.503966 IP 10.0.0.73.25 > 10.0.0.5.56150: Flags [S.], seq 940845167, ack 3129724328, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2075520144 ecr 25670986], length 0
16:50:20.503979 IP 10.0.0.5.56150 > 10.0.0.73.25: Flags [R], seq 3129724328, win 0, length 0So the redirect and NAT rule works as expected, but the reply is never returned to telnet which, for some reason, sends the request on lo0 (even when a specific source address is specified)... Or something?
So could anyone explain what's going on here? And it is even possible to do what I want?