Originating IP from Mailserver Round Robin

onyxrev

onyxrev

I've got a server set up as a mail server. The server also has a few other services on it assigned to separate IP addresses.

I'm having some difficulty getting my mail SPF records to consistently pass because sometimes for some recipients the mail appears to be coming from the non-mailserver IP addresses.

The mail server consists of Postfix and Dovecot and is running in a jail. The webservers are running in separate jails. The 'master' server running the jails has NAT running to the jails via PF. This is all on FreeBSD 9.

The configs of the mail server do not even mention any other IPs.

Here's the output of ifconfig:

Code: 
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether 52:54:00:9f:75:60
	inet [ip address 1] netmask 0xffffff00 broadcast [subnet]
	inet6 [ipv6 address 1] prefixlen 64 scopeid 0x2 
	inet [ip address 2] netmask 0xffffffff broadcast [ip address 2]
	inet [ip address 3] netmask 0xffffffff broadcast [ip address 3]
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 10.1.1.1 netmask 0xffffff00 
	inet 10.1.1.2 netmask 0xffffffff 
	inet 10.1.1.3 netmask 0xffffffff 
	inet 10.1.1.4 netmask 0xffffffff 
	inet 10.1.1.6 netmask 0xffffffff 
	inet 10.1.1.7 netmask 0xffffffff 
	inet 10.1.1.8 netmask 0xffffffff 
	inet 10.1.1.10 netmask 0xffffffff 
	inet 10.1.1.11 netmask 0xffffffff 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Here's the networking parts of rc.conf:

Code: 
ifconfig_re0="inet [ip address 1]   netmask 255.255.255.0"

# jails external interfaces
ifconfig_re0_alias0="inet [ip address 2] netmask 255.255.255.255" # webserver 2
ifconfig_re0_alias1="inet [ip address 3] netmask 255.255.255.255" # unused

# jail aliases
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.1.1.1 netmask 255.255.255.0"          # webserver1
ifconfig_lo1_alias0="inet 10.1.1.2 netmask 255.255.255.255" # database1
ifconfig_lo1_alias1="inet 10.1.1.3 netmask 255.255.255.255" # mail & jabber
ifconfig_lo1_alias2="inet 10.1.1.4 netmask 255.255.255.255" # loadbalancer1
ifconfig_lo1_alias3="inet 10.1.1.6 netmask 255.255.255.255" # webserver 2
ifconfig_lo1_alias4="inet 10.1.1.7 netmask 255.255.255.255" # git
ifconfig_lo1_alias5="inet 10.1.1.8 netmask 255.255.255.255" # webserver 3
ifconfig_lo1_alias6="inet 10.1.1.10 netmask 255.255.255.255" # webserver 4

Here are relevant routing bits of pf.conf:

Code: 
##############
# Interfaces #
##############

ext_if="re0"
ext_ip="[ip address 1]"
int_if="lo1"
jailnet=$int_if:network

#########
# Ports #
#########

web_ports     = "{ http, https }"
mail_ports    = "{ imaps, imap, smtps, submission, smtp-alt, smtp, pop3s, pop3 }"
jabber_ports  = "{ xmpp-client, xmpp-server }"
torrent_ports = "{ 49164, 6881 }"
icmp_types    = "echoreq"
# these IPs are invalid "Internet" IPs, reserved for private networks
non_routable  = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8, 169.254.0.0/16,
 192.0.2.0/24, 0.0.0.0/8, 240.0.0.0/4 }"

##############
# Jails Defs #
##############

loadbalancer1_int = "10.1.1.4"

mailserver_int = "10.1.1.3"

webserver2_ext = "[ip address 2]"
webserver2_int = "10.1.1.6"

set skip on lo
scrub in

#######
# NAT #
#######

## all
# allow out from internal nat interface
nat on $ext_if from $jailnet to any -> ($ext_if)
nat on $int_if from $jailnet to any -> ($int_if)

## loadbalancer1
# handles most web traffic
# forward web ports so they don't go to master but instead to loadbalancer1
rdr pass on $ext_if inet proto tcp from any to $ext_ip port $web_ports -> $loadbalancer1_int 

## Mail
# handles all mail traffic
# forward mail ports so they don't go to master but instead to mailserver
rdr pass on $ext_if inet proto tcp from any to $ext_ip port $mail_ports -> $mailserver_int
rdr pass on $ext_if inet proto tcp from any to $ext_ip port $jabber_ports -> $mailserver_int

## Webserver 2
# ecommerce SSL dictates webserver get its own external IP
# allow communication from webserver2 to the outside and the outside to webserver2 via its external ip
#binat on $ext_if from $webserver2_int to any -> $webserver2_ext
rdr pass on $ext_if inet proto tcp from any to $webserver2_ext port $web_ports -> $webserver2_int

To me it seems like the IP aliasing is the most likely culprit. What I'm seeing is that sometimes the emails will originate from [IP address 1] but sometimes they will originate from [IP address 2] or [IP address 3]. I only want mail originating from [IP address 1].

Is 'alias' doing something subtle here that might cause the outgoing IP to randomize from the pool despite the mail configs and PF routing only specifying one IP?

Thank you, I really appreciate it.
 
Not sure if this is going to work but try something like this:
Code: 
nat on $ext_if from $mailserver_int to any -> ($ext_if:0)
 
SirDice said:
Not sure if this is going to work but try something like this:
Code: 
nat on $ext_if from $mailserver_int to any -> ($ext_if:0)
Click to expand...

Worth a try but didn't help.
 
SirDice said:
Not sure if this is going to work but try something like this:
Code: 
nat on $ext_if from $mailserver_int to any -> ($ext_if:0)
Click to expand...

You were absolutely on the right path, though. Check out the expansion of my rules when running pctl -vnf /etc/pf.conf:

Code: 
nat on re0 inet from 10.1.1.0/24 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.2 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.3 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.4 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.6 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.7 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.8 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.10 to any -> (re0) round-robin
nat on re0 inet from 10.1.1.11 to any -> (re0) round-robin
nat on lo1 inet from 10.1.1.0/24 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.2 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.3 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.4 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.6 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.7 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.8 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.10 to any -> (lo1) round-robin
nat on lo1 inet from 10.1.1.11 to any -> (lo1) round-robin

It's round-robin because it's... round-robin. For now I'm going to do individual nat rules for each jail. I'm on IRC with #pf to see if there's a way to always redirect out to the original destination IP.
 
The PF man says that round-robin is always on if the outbound IPs are a pool. So I think that means if you want to ensure the outgoing IP is always the same you need individual rules. So in my case:

Code: 
nat on $ext_if from $jailnet to any -> ($ext_if)
becomes

Code: 
nat on re0 inet from {10.1.1.3, 10.1.1.4} to any -> $ext_ip
nat on re0 inet from {10.1.1.6, 10.1.1.7} to any -> $webserver2_ext
or whatever cleans that up with macros.

That works. The outbound IP is consistent with those rules. It was PF the whole time.
 
SirDice said:
Not sure if this is going to work but try something like this:
Code: 
nat on $ext_if from $mailserver_int to any -> ($ext_if:0)
Click to expand...

Actually you were correct. The rule has to be placed before the others because it's first-match.

So here's what I ended up with:

Code: 
# mailserver *must* have a specific outbound IP
nat on $ext_if from $mailserver_int  to any -> $ext_ip

## all
# allow out from internal nat interface
# NOTE: these rules end up round-robin on all external IPs!
#       if you need specific outbound IPs, make a rule above
nat on $ext_if from $jailnet to any -> ($ext_if)
nat on $int_if from $jailnet to any -> ($int_if)
 
You must log in or register to reply here.
Back
Top