Solved Example setup for email behind VPS, VPN, DSL (originally a NAT loopback problem)

S

sadaszewski

Hi,

I have OpenVPN in a jail on one machine and natively on another. Both machines are VMs running FreeBSD 13. I am trying to redirect a port from one machine to the other without losing the source address. This works, as the packets easily traverse where they need to go, as verified with tcpdump on BOTH ENDS. It is well thought out (I think) with a second routing table and a separate IP in addition to the one assigned by VirtualBox DHCP to make sure that packets end up on correct devices. And they do. However, the client unexpectly breaks up the connection with a TCP RST. The setup and details look the following:

CLIENT

ifconfig
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 08:00:27:76:06:22
hwaddr 08:00:27:76:06:22
inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600000<RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.1.0 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: lo
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.8.0.2 --> 10.8.0.1 netmask 0xff000000
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: tun
netstat -rn -F 0
Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.2.2           UGS         em0
10.0.2.0/24 link#1 U em0
10.0.2.15          link#1             UHS         lo0
10.8.0.1           link#4             UH         tun0
10.8.0.2           link#4             UHS         lo0
127.0.0.1          link#2             UH          lo0
127.0.1.0          link#3             UH          lo1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16 ::1 UGRS lo0
netstat -rn -F 1
Code: 
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.8.0.1           UGS        tun0
[URL='http://10.0.2.0/24']10.0.2.0/24[/URL] link#1 U em0
10.8.0.1           link#4             UH         tun0
127.0.0.1          link#2             UH          lo0
127.0.1.0          link#3             UH          lo1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:[URL='http://0.0.0.0/96']0.0.0.0/96[/URL] ::1 UGRS lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
ff02::/16                         ::1                           UGRS        lo0
/boot/loader.conf
Code: 
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
net.fibs="2"
/etc/pf.conf
Code: 
pass on lo0 all
pass on em0 all
pass on tun0 all
crontab (root)
Code: 
@every_minute if [ "`ifconfig tun0 | grep inet | cut -d' ' -f2`" != "10.8.0.2" ] ; then ifconfig tun0 inet 10.8.0.2 10.8.0.1 && route add default 10.8.0.1 -fib 1; fi >/dev/null 2>&1
/etc/jail.conf
Code: 
70681eb {
path = '/focker/jails/70681eb';
persist;
interface = 'lo1';
ip4.addr = '127.0.1.0';
mount.devfs;
exec.clean;
host.hostname = 'ovpn-cli';
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
exec.prestart = 'cp /etc/resolv.conf /focker/jails/70681eb/etc/resolv.conf';
}

e1aca4b {
path = '/focker/jails/e1aca4b';
persist;
interface = 'tun0';
ip4.addr = 'tun0|10.8.0.2 10.8.0.1';
mount.devfs;
exec.clean;
host.hostname = 'mail';
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
exec.fib = 1;
exec.prestart = 'cp /etc/resolv.conf /focker/jails/e1aca4b/etc/resolv.conf && route add default 10.8.0.1 -fib 1';
}
JAIL 1 - ovpn-cli - (ON THE CLIENT)

/usr/local/etc/openvn/openvpn.conf
Code: 
proto udp4
dev tun0
remote 10.0.2.2
port 1197
ifconfig 10.8.0.2 10.8.0.1
secret /usr/local/etc/openvpn/static.key
cipher aes-256-cbc
ifconfig-noexec
keepalive 10 60
Starting manually from the host:
Code: 
# service jail start 70681eb
# jexec 70681e
jail # openvpn /usr/local/openvpn/openvpn.conf
JAIL 2 - mail - (ON THE SERVER)

Jail is started manually on the host:
Code: 
# service jail start e1aca4b
# setfib 1 jexec e1aca4b
jail # nc -l 2525
SERVER

ifconfig
Code: 
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4810098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
        ether 08:00:27:ed:9b:4f
        inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
        inet 10.0.3.15 netmask 0xffffffff broadcast 10.0.3.15
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8149<UP,LOOPBACK,RUNNING,PROMISC,MULTICAST> metric 0 mtu 16384
options=680000<LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
netstat -rn
Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.2.2           UGS         em0
[URL='http://10.0.2.0/24']10.0.2.0/24[/URL] link#1 U em0
10.0.2.15          link#1             UHS         lo0
10.0.3.15          link#1             UH          em0
127.0.0.1          link#2             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UHS         lo0
::ffff:[URL='http://0.0.0.0/96']0.0.0.0/96[/URL] ::1 UGRS lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16 ::1 UGRS lo0
/usr/local/etc/openvpn/openvpn.conf
Code: 
proto udp4
dev tun0
port 1197
ifconfig 10.8.0.1 10.8.0.2
secret /usr/local/etc/openvpn/static.key
cipher aes-256-cbc
keepalive 10 60
/etc/pf.conf
Code: 
rdr pass inet proto tcp from any to 10.0.3.15 port 2525 -> 10.8.0.2 port 2525

pass on em0 all
pass on lo0 all
pass on tun0 all
I start OpenVPN manually:
Code: 
# openvpn /usr/local/etc/openvpn/openvpn.conf
Connectivity works, e.g. ssh 10.8.0.2 from the server or ssh 10.8.0.1 from the client. All perfect

Then I try to connect to 10.0.3.15:2525 which should redirect me to 10.8.0.2:2525 like this:

# telnet 10.0.3.15 2525

I have verified with tcpdump on BOTH ends that the following exchange takes place:
Code: 
12:53:19.632773 IP (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.0.3.15.26360 > 10.8.0.2.2525: Flags , cksum 0x3d15 (correct), seq 2149400320, win 65535, options [mss 1460,sackOK,eol], length 0
12:53:19.636213 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.8.0.2.2525 > 10.0.3.15.26360: Flags [S.], cksum 0x6969 (correct), seq 1358332690, ack 2149400321, win 65535, options [mss 1350,sackOK,eol], length 0
12:53:19.636223 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
10.0.3.15.26360 > 10.8.0.2.2525: Flags [R], cksum 0x68d4 (correct), seq 2149400321, win 0, length 0
The setup is on two VirtualBox instances and port 1197 on the server is redirected so that 10.0.2.2 (IP of the host PC as visible from the guest) at port 1197 redirects to the guest port 1197.

I did ifconfig -rxcsum -txcsum on lo0 and em0 on both VMs as otherwise I was getting the famous checksum errors with this redirect.

Is there a reason why the client suddenly replies with TCP RST? Could someone who is experienced share an opinion please? Thank you in advance.

Kind regards,

--
Stanislaw
 
Ok, mystery solved - it's just a simple case of missing (NAT reflection / NAT hairpinning / NAT loopback / other names by which we know the mechanism of redirecting packets from internal IPs targeted at an external IP to internal hosts). It works from the outside to put it simply. I will leave the config up there for reference, as otherwise it is a nice way to set up a mail service on your own computer connected to a residential DSL with a VPN to a static IP VPS, without losing the real source IP addresses of incoming connections.
 
You must log in or register to reply here.
Back
Top