NAT+pf+multi gateway issue

PaulWebster

PaulWebster

Good day all,

I have a working home network that has the following layout:

[Clients (172.31.33.2-172.31.33.200]
|
[Switch||Wireless AP]
|
[Gateway (172.31.33.1,PUBLIC_IP)]
|
{internet}

miniupnpd is enabled as well as a few other bits of tinsel, but all in all works perfectly..

Now the issue;

Of those clients, 2 are xboxs and because of the security measures I have in place they do not work correctly with some multiplayer servers and because there is two xboxs I cannot manually forward the required ports (as there not dynamic) that would only work for 1 of them.

So genius idea, cheap VPS <10ms away lets use that as a public gateway for just the xbox's (1 each)

Now, first of all I assigned a dhcp-host line for the xbox's so there given the ips 172.31.33.254 and 172.31.33.243 (respective, from here on out I will just concentrate on the .254)

So the first thing I did was create a bridge between the freebsd gateway and the VPS, this gave my gateway (we will call it GW1) the ip of 10.0.0.2 and the VPS the ip of 10.0.0.1; I did try this with gre tunnels, gif tunnels, tun0 tunnels; in hte end I decided on using 'tincd' with no encryption and a compression level of '1'; meaning fastest zlib compression.

With that in place from GW1 pinging GW2 I get:

root@GW1:~ # ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=157.440 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=134.502 ms

Note the high latency, this is not a problem there is a large upload in progress for another service.

10.0.0.2(GW1) and 10.0.0.1(GW2) are pingable by all 172.31.33.1/24 clients

as a note on GW1 the NAT rule that works for the clients in the .1-200 range is thus:
nat on $ext_if from $localnet to any -> ($ext_if)

$localnet is a { array } of the ips in range of 1-200, so that the xbox that is currently on .254 is not being NAT'd anywhere at the moment.

As a little more info from GW2 the xbox has this ping (now the network is settled down a little):


root@GW2:~ # ping 172.31.33.254
PING 172.31.33.254 (172.31.33.254): 56 data bytes
64 bytes from 172.31.33.254: icmp_seq=0 ttl=63 time=327.917 ms
64 bytes from 172.31.33.254: icmp_seq=1 ttl=63 time=40.764 ms
64 bytes from 172.31.33.254: icmp_seq=2 ttl=63 time=42.158 ms
64 bytes from 172.31.33.254: icmp_seq=3 ttl=63 time=24.336 ms

So as you can see for the sake of the link it all seems happy, if I was to redirect tcp/udp ports 1-65535 in to 172.31.33.254 from the GW2 for all effects and purposes it would be a DMZ of kinds.

The issue comes with the outbound path from GW1.

How do I tell GW1 that packs from the xbox(172.31.33.254) should be routed out via 10.0.0.1 (GW2)?
 
Also ... speedtest from GW2:
Code: 
root@GW2:~ # ./speedtest-cli
Retrieving speedtest.net configuration...
Testing from LoveServers Ltd (185.157.232.30)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Cybersmart Pty Ltd (London) [2.56 km]: 18.143 ms
Testing download speed................................................................................
Download: 327.00 Mbit/s
Testing upload speed....................................................................................................
Upload: 201.94 Mbit/s

and GW1:

Code: 
root@GW1:~ # ./speedtest-cli
Retrieving speedtest.net configuration...
Testing from Virgin Media (86.5.192.180)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Synergy-UK (Sheffield) [48.16 km]: 33.736 ms
Testing download speed................................................................................
Download: 36.05 Mbit/s
Testing upload speed....................................................................................................
Upload: 5.28 Mbit/s

But if I do a speed test forcing a connection through the VPN (by starting a socks proxy server on GW2) and telling .254 to use it for firefox or so, the speed is DIRE slow.

if I setup a socks proxy on GW2 and access it outside of the VPN (direct over the net) I get 100mbit+
 
Just to clarify the question;

How do I get it so that all packets from 172.31.33.254(xbox) get routed out via GW2(10.0.0.2); without the massive speed penalities that seem to occur when using a vpn, as I said in the previous post with a 'non vpn' socks server from the vps I get 100mbit+ download, through the vpn I get <0.1mbps
 
A little update, a thank you to 'rgrimes' on freenode #freebsd, who mentioned the way to tackle this is to use 'fibs' or 'multiple routing tables'

The new config is done with 'setfib 1 route add default 10.0.0.2', which means that 'routing table 1' now has the default route of 10.0.0.2 which is perfect, also I know that 10.0.0.1 and 10.0.0.2 appear to have swapped around, this is because I went from using tincd/tun to a straight gre tunnel, so now GW1 is 10.0.0.1 GW2 is 10.0.0.2.

Anyhow this rule was added to GW1's PF:
pass in quick on $int_if from $josh_xbox rtable 1 keep state

Which means when a packet is received from 172.31.33.254 (josh_xbox) use routing table 1, now this appears to have SOME WHAT Worked. as now on GW2 on gre0 we have:


root@GW2:~ # tcpdump -i gre0 -vvv icmp
tcpdump: listening on gre0, link-type NULL (BSD loopback), capture size 262144 bytes
22:16:33.025339 IP (tos 0x0, ttl 63, id 18432, offset 0, flags [DF], proto ICMP (1), length 84)
172.31.33.254 > google-public-dns-a.google.com: ICMP echo request, id 11317, seq 1, length 64
22:16:34.029455 IP (tos 0x0, ttl 63, id 18433, offset 0, flags [DF], proto ICMP (1), length 84)
172.31.33.254 > google-public-dns-a.google.com: ICMP echo request, id 11317, seq 2, length 64
22:16:35.028421 IP (tos 0x0, ttl 63, id 18434, offset 0, flags [DF], proto ICMP (1), length 84)
172.31.33.254 > google-public-dns-a.google.com: ICMP echo request, id 11317, seq 3, length 64

Which is our xbox trying to ping 8.8.8.8!

Now the problem ...

the src is still 172.31.33.254, so when the ICMP leaves vtnet0 (GW2's public interface) via this rule:

binat on $ext_if from $DMZ_host to any -> 185.157.232.30/32

it never gets translated from 172.31.33.254 to 185.157.232.30 so gets dropped at the next hop.

I have NO IDEA what to do from here so any hints are more than welcome :)
 
And we are working!

turns out I forgot to enable pf on GW2, I am using the following rules:

Code: 
ext_if="vtnet0"
int_if="gre0"
DMZ_host="86.5.192.180"

set require-order yes

nat on $ext_if from !$ext_if to any -> ($ext_if)

pass in quick on $ext_if inet proto 47 from $DMZ_host to any
pass out quick on $ext_if inet proto 47 from any to $DMZ_host

Though the speed is dreadful, some sites (especially SSL) do not work .. so kinda half way there; gre0 on both hosts is mtu 1476 which seems reasonable.
 
Lets take a speed test from Every level;

GW2: {
fetch http://speed.hetzner.de/100MB.bin
} 2800kBps

GW1: {
ssh -D 10.0.0.2:8080 USER@10.0.0.2
SOCKS_SERVER=110.0.0.2:8080 wget http://speed.hetzner.de/100MB.bin
} 2.50MB/s

Xbox(using a linux laptop in place of to be able to do anything useful) {
chronos@localhost ~/Downloads $ wget http://speed.hetzner.de/100MB.bin#
--2017-04-03 23:04:52-- http://speed.hetzner.de/100MB.bin
Resolving speed.hetzner.de... 88.198.248.254
Connecting to speed.hetzner.de|88.198.248.254|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: '100MB.bin'

100MB.bin 0%[ ] 9.57K --.-KB/s eta 6d 4h
} ..... ta da
 
To add to the following speed test ... I wondered what would happen if I scp'd a file STRAIGHT to GW2 from the 'xbox'

Xbox->GW1
chronos@localhost ~/Downloads $ scp FreeBSD-11.0-RELEASE-amd64-disc1.iso paul@10.0.0.1:
Password for paul@bsd.dlink.com:
FreeBSD-11.0-RELEASE-amd64-disc1.iso 9% 60MB 11.5MB/s 00:51 ETA

Xbox->GW2
chronos@localhost ~/Downloads $ scp FreeBSD-11.0-RELEASE-amd64-disc1.iso paul@10.0.0.2:
Password for paul@endpoint.uk.tmp.group:
FreeBSD-11.0-RELEASE-amd64-disc1.iso 3% 24MB 741.6KB/s 14:32 ETA

Xbox->Another one of my servers
chronos@localhost ~/Downloads $ scp FreeBSD-11.0-RELEASE-amd64-disc1.iso paul@hetzner.tmp.group:
Password for paul@hetzner.tmp.group:
FreeBSD-11.0-RELEASE-amd64-disc1.iso 1% 11MB 1.2MB/s 08:51 ETA

Xbox retrieve the 100M file again!
chronos@localhost ~/Downloads $ wget http://speed.hetzner.de/100MB.bin#
--2017-04-03 23:36:36-- http://speed.hetzner.de/100MB.bin
Resolving speed.hetzner.de... 88.198.248.254
Connecting to speed.hetzner.de|88.198.248.254|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: '100MB.bin'

100MB.bin 0%[ ] 6.86K 4.05KB/s


so yeah ..... WTF comes to mind, here is the full pf for GW1
Code: 
# Macros
ext_if=igb0
int_if=igb1
localnet = "{ 172.31.33.1/32, 172.31.33.2/32, 172.31.33.3/32, 172.31.33.4/32, 172.31.33.5/32, 172.31.33.6/32, 172.31.33.7/32, 172.31.33.8/32, 172.31.33.9/32, 172.31.33.10/32, 172.31.33.11/32, 172.31.33.12/32, 172.31.33.13/32, 172.31.33.1

josh_xbox="172.31.33.254/32"
josh_if="gre0"
#josh_gateway="10.0.0.1"
josh_gateway="185.157.232.30"
josh_peer="3.3.3.3"

tcp_services = "{ ssh, smtp, domain, www, pop3, auth, pop3s, 5901 }"
udp_services = "{ domain }"

# Global rules
set skip on lo0
#set skip on gif0
scrub in all

# NAT and redirection
nat on $ext_if from $localnet to any -> ($ext_if)
#binat on $josh_if from $josh_xbox to any -> $josh_gateway
#nat on $josh_if from $josh_xbox to any -> ($josh_if)

#rdr pass on $int_if inet proto {tcp,udp} from $josh_xbox to any -> $josh_gateway

rdr-anchor "miniupnpd"

# Tables and sets
table <bruteforce> persist
table <blocked> persist

# Filtering rules (Quick first)

# SSH, DNS, DHCP
block quick on $ext_if proto udp from any to any port 67
pass in quick on $int_if proto tcp from 172.31.33.1/24 to 172.31.33.1/32 port 22
pass in quick on $int_if proto {tcp,udp} from 172.31.33.1/24 to 172.31.33.1/32 port 53
pass in quick on $int_if proto udp from any to 172.31.33.1/32 port 63

# Pass out the xbox traffic
pass in quick on $int_if from $josh_xbox rtable 1 keep state
#pass in quick on $int_if route-to ($josh_if $josh_gateway) from $josh_xbox to any
#pass in quick on $int_if route-to ($josh_if $josh_gateway) from $josh_xbox to any no state flags any
# pass in quick on $int_if route-to ($josh_if $josh_gateway) from $josh_xbox to any
#pass in quick on $int_if route-to "185.157.232.30" from $josh_xbox to any

# Release GRE and QUICK release the protocol
pass in quick on $ext_if inet proto 47 from $josh_peer to any
pass out quick on $ext_if inet proto 47 from any to $josh_peer
pass quick on $josh_if

# Anti asshole rule
block log quick on $ext_if from <bruteforce>
block log quick on $int_if from <blocked>

# Filtering rules generic

# Drop icmp inbound
#block in on $ext_if inet proto icmp all
#pass in on $ext_if inet proto icmp all icmp-type echoreq keep state

# Watch for assholes on ssh and others
pass in on $ext_if proto tcp from any to any port $tcp_services \
    flags S/SA keep state \
    (max-src-conn 100, max-src-conn-rate 15/5, \
    overload <bruteforce> flush global)

# Disallow external access to minupnpd
block on $ext_if proto tcp from any to port 33333
block on $ext_if proto udp from any to port 1900

anchor "miniupnpd"
 
Wget command:

(precise)root@localhost:/home/paul# wget http://speed.hetzner.de/100MB.bin
--2017-04-04 00:17:51-- http://speed.hetzner.de/100MB.bin
Resolving speed.hetzner.de (speed.hetzner.de)... 88.198.248.254, 2a01:4f8:0:59ed::2
Connecting to speed.hetzner.de (speed.hetzner.de)|88.198.248.254|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `100MB.bin.1'

0% [ ] 5,192 1.03K/s eta 27h 44m

TCPDump (tcpdump -i wlan0 -vvv port 80):

Code: 
0:17:45.867532 IP (tos 0x0, ttl 64, id 19495, offset 0, flags [DF], proto TCP (6), length 332)
   172.31.33.254.47841 > lhr25s10-in-f3.1e100.net.http: Flags [P.], cksum 0x0bc5 (correct), seq 2876460209:2876460489, ack 2656081017, win 115, options [nop,nop,TS val 12557701 ecr 702749648], length 280: HTTP, length: 280
        GET /generate_204 HTTP/1.1
        Host: www.gstatic.com
        Connection: keep-alive
        Pragma: no-cache
        Cache-Control: no-cache
        User-Agent: Mozilla/5.0 (X11; CrOS x86_64 9000.91.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.110 Safari/537.36
        Accept-Encoding: gzip, deflate
      
00:17:45.894689 IP (tos 0x0, ttl 55, id 7216, offset 0, flags [none], proto TCP (6), length 135)
    lhr25s10-in-f3.1e100.net.http > 172.31.33.254.47841: Flags [P.], cksum 0x028b (correct), seq 1:84, ack 280, win 399, options [nop,nop,TS val 702760791 ecr 12557701], length 83: HTTP, length: 83
        HTTP/1.1 204 No Content
        Content-Length: 0
        Date: Mon, 03 Apr 2017 23:17:46 GMT
      
00:17:45.894737 IP (tos 0x0, ttl 64, id 19496, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.33.254.47841 > lhr25s10-in-f3.1e100.net.http: Flags [.], cksum 0x1229 (correct), seq 280, ack 84, win 115, options [nop,nop,TS val 12557728 ecr 702760791], length 0
00:17:51.364563 IP (tos 0x0, ttl 64, id 29594, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.33.254.52671 > speedtest.your-server.de.http: Flags , cksum 0x75e7 (correct), seq 9[CODE]
05757028, win 14600, options [mss 1460,sackOK,TS val 12563198 ecr 0,nop,wscale 7], length 0
00:17:51.403445 IP (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 48)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [S.], cksum 0xd100 (correct), seq 3091862883, ack 905757029, win 65535, options [mss 1460,nop,wscale 13], length 0
00:17:51.403513 IP (tos 0x0, ttl 64, id 29595, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xfc5e (correct), seq 1, ack 1, win 115, length 0
00:17:51.403513 IP (tos 0x0, ttl 64, id 29595, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xfc5e (correct), seq 1, ack 1, win 115, length 0
00:17:51.403735 IP (tos 0x0, ttl 64, id 29596, offset 0, flags [DF], proto TCP (6), length 165)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [P.], cksum 0x349d (correct), seq 1:126, ack 1, win 115, length 125: HTTP, length: 125
GET /100MB.bin HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: speed.hetzner.de
Connection: Keep-Alive

00:17:51.442790 IP (tos 0x0, ttl 52, id 44387, offset 0, flags [DF], proto TCP (6), length 40)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0xf9f1 (correct), seq 1, ack 126, win 611, length 0
00:17:51.791260 IP (tos 0x0, ttl 52, id 44398, offset 0, flags [DF], proto TCP (6), length 1402)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0x00ad (correct), seq 1:1363, ack 126, win 611, length 1362: HTTP, length: 1362
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 03 Apr 2017 23:17:51 GMT
Content-Type: application/octet-stream
Content-Length: 104857600
Last-Modified: Tue, 08 Oct 2013 11:48:13 GMT
Connection: keep-alive
ETag: "5253f0fd-6400000"
Accept-Ranges: bytes

00:17:51.791326 IP (tos 0x0, ttl 64, id 29597, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xf67a (correct), seq 126, ack 1363, win 136, length 0
00:17:52.508022 IP (tos 0x0, ttl 52, id 44401, offset 0, flags [DF], proto TCP (6), length 1402)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0xef4d (correct), seq 1363:2725, ack 126, win 611, length 1362: HTTP

00:17:52.508082 IP (tos 0x0, ttl 64, id 29598, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xf113 (correct), seq 126, ack 2725, win 157, length 0
00:17:53.736703 IP (tos 0x0, ttl 52, id 44404, offset 0, flags [DF], proto TCP (6), length 1402)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0xe9fb (correct), seq 2725:4087, ack 126, win 611, length 1362: HTTP
00:17:53.736810 IP (tos 0x0, ttl 64, id 29599, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xebac (correct), seq 126, ack 4087, win 178, length 0
00:17:55.784470 IP (tos 0x0, ttl 52, id 44407, offset 0, flags [DF], proto TCP (6), length 1402)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0xe4a9 (correct), seq 4087:5449, ack 126, win 611, length 1362: HTTP
00:17:55.784529 IP (tos 0x0, ttl 64, id 29600, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [.], cksum 0xe644 (correct), seq 126, ack 5449, win 200, length 0
00:17:57.153496 IP (tos 0x0, ttl 64, id 29601, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [F.], cksum 0xe643 (correct), seq 126, ack 5449, win 200, length 0
00:17:57.229932 IP (tos 0x0, ttl 52, id 44410, offset 0, flags [DF], proto TCP (6), length 40)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0x9a2c (correct), seq 24517, ack 127, win 611, length 0
00:17:59.880318 IP (tos 0x0, ttl 52, id 44411, offset 0, flags [DF], proto TCP (6), length 1402)
speedtest.your-server.de.http > 172.31.33.254.52671: Flags [.], cksum 0xdf56 (correct), seq 5449:6811, ack 127, win 611, length 1362: HTTP
00:17:59.880375 IP (tos 0x0, ttl 64, id 47217, offset 0, flags [DF], proto TCP (6), length 40)
172.31.33.254.52671 > speedtest.your-server.de.http: Flags [R], cksum 0xca0e (correct), seq 905757155, win 0, length 0[/CODE]
 
GW1<->GW2 iperf

GW2:
root@endpoint:~ # iperf3 -sp 2000
-----------------------------------------------------------
Server listening on 2000
-----------------------------------------------------------
Accepted connection from 10.0.0.1, port 21384
[ 5] local 10.0.0.2 port 2000 connected to 10.0.0.1 port 21385
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 640 KBytes 5.24 Mbits/sec
[ 5] 1.00-2.00 sec 732 KBytes 6.00 Mbits/sec
[ 5] 2.00-3.00 sec 729 KBytes 5.97 Mbits/sec
[ 5] 3.00-4.00 sec 722 KBytes 5.92 Mbits/sec
[ 5] 4.00-5.00 sec 722 KBytes 5.92 Mbits/sec
[ 5] 5.00-6.00 sec 729 KBytes 5.97 Mbits/sec
[ 5] 6.00-7.00 sec 725 KBytes 5.94 Mbits/sec
[ 5] 7.00-8.00 sec 730 KBytes 5.98 Mbits/sec
[ 5] 8.00-9.00 sec 725 KBytes 5.94 Mbits/sec
[ 5] 9.00-10.00 sec 725 KBytes 5.94 Mbits/sec
[ 5] 10.00-10.20 sec 144 KBytes 5.91 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.20 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.20 sec 7.15 MBytes 5.88 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 2000
-----------------------------------------------------------


GW1:
root@bsd:/usr/ports # iperf3 -c 10.0.0.2 -p 2000
Connecting to host 10.0.0.2, port 2000
[ 4] local 10.0.0.1 port 21385 connected to 10.0.0.2 port 2000
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 734 KBytes 6.02 Mbits/sec 5 51.2 MBytes
[ 4] 1.00-2.00 sec 728 KBytes 5.96 Mbits/sec 0 77.8 MBytes
[ 4] 2.00-3.00 sec 768 KBytes 6.29 Mbits/sec 0 97.2 MBytes
[ 4] 3.00-4.00 sec 722 KBytes 5.92 Mbits/sec 0 113 MBytes
[ 4] 4.00-5.00 sec 744 KBytes 6.10 Mbits/sec 0 127 MBytes
[ 4] 5.00-6.00 sec 749 KBytes 6.13 Mbits/sec 0 140 MBytes
[ 4] 6.00-7.00 sec 730 KBytes 5.98 Mbits/sec 0 152 MBytes
[ 4] 7.00-8.00 sec 752 KBytes 6.16 Mbits/sec 0 163 MBytes
[ 4] 8.00-9.00 sec 722 KBytes 5.92 Mbits/sec 0 173 MBytes
[ 4] 9.00-10.00 sec 741 KBytes 6.07 Mbits/sec 0 182 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 7.22 MBytes 6.05 Mbits/sec 5 sender
[ 4] 0.00-10.00 sec 7.15 MBytes 6.00 Mbits/sec receiver

iperf Done.
 
As an aside, the traceroute and ICMP type lookups from the xbox/laptop to <anywhere> show a perfect reasonable return based on availible bandwidth; 21ms to 10.0.0.2 etc..
 
You must log in or register to reply here.
Back
Top