PF PF port forwarding

Hello,

I am new to pf and want to learn how to use it but encounter some problems with port forwarding.

These are my hosts :

Host A : 89.28.16.166
Host B : 128.209.98.106
Host C : 89.85.24.118

From host A, I would like to ssh (port 887) to host B and be redirected to host C (port 888).

hostA> ssh 128.209.98.106 -p 887 -> 89.85.24.118 port 888

Host B :
/etc/rc.conf
Code:
gateway_enable="YES"

/etc/sysctl.conf
Code:
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet6.ip6.forwarding=1


/etc/pf.conf
Code:
set skip on lo0
rdr pass on vtnet0 proto tcp from any to any port 887 -> 89.85.24.118 port 888

I can ssh from host B to host C on port 888 :

# ssh 89.85.24.118 -p 888

Host C :

Accept all traffic.


On host B, I can see incoming traffic from host A on port 887

Code:
# tcpdump -nni vtnet0 'port 887'
11:11:15.614859 IP 89.28.16.166.36664 > 128.209.98.106.887: Flags [ S ], seq 3656692557, win 29200, options [mss 1452,sackOK,TS val 281541214 ecr 0,nop,wscale 7], length 0

On host B, I can see outgoing forwarded traffic from host A to host C :

Code:
# tcpdump -nni vtnet0 'port 888'
11:16:10.835463 IP 89.28.16.166.36875 > 89.85.24.118.888: Flags [ S ], seq 2038275656, win 29200, options [mss 1452,sackOK,TS val 281615020 ecr 0,nop,wscale 7], length 0

But on host C, I can't see any incoming traffic :

Code:
# tcpdump -i vtnet0 'port 888'
...

What do I wrong ? Can somebody help me ?

Thanks a lot...


Peter.
 
This comes up quite frequently:
Code:
     Translation rules apply only to packets that pass through the specified
     interface, and if no interface is specified, translation is applied to
     packets on all interfaces.  For instance, redirecting port 80 on an
     external interface to an internal web server will only work for connec-
     tions originating from the outside.  Connections to the address of the
     external interface from local hosts will not be redirected, since such
     packets do not actually pass through the external interface.  [b]Redirec-
     tions cannot reflect packets back through the interface they arrive on,
     they can only be redirected to hosts connected to different interfaces or
     to the firewall itself.[/b]
From pf.conf(5).
 
Your rdr rule is way too broad becuse it matches traffic destined to all addresses, rewrite it to this:

Code:
rdr pass on vtnet0 proto tcp from any to vtnet0 port 887 -> 89.85.24.118 port 888
 
Your rdr rule is way too broad becuse it matches traffic destined to all addresses, rewrite it to this:

Code:
rdr pass on vtnet0 proto tcp from any to vtnet0 port 887 -> 89.85.24.118 port 888

I tried it but it didn't work neither...
 
This comes up quite frequently:
Code:
     Translation rules apply only to packets that pass through the specified
     interface, and if no interface is specified, translation is applied to
     packets on all interfaces.  For instance, redirecting port 80 on an
     external interface to an internal web server will only work for connec-
     tions originating from the outside.  Connections to the address of the
     external interface from local hosts will not be redirected, since such
     packets do not actually pass through the external interface.  Redirec-
     tions cannot reflect packets back through the interface they arrive on,
     they can only be redirected to hosts connected to different interfaces or
     to the firewall itself.
From pf.conf(5).


I didn't know that. It make sense...

But what if the forwarding host have two network adapters ?

I tried this out :

/etc/pf.conf

Code:
extNic="vtnet0"
extIp="128.209.98.106"

intNic="vtnet1"
intIp="10.129.23.150"

rdr pass on $extNic proto tcp from any to $extIp port 701 -> 10.129.23.125 port 22

When I ssh to 128.209.98.106 port 701 I can see this in tcpdump(1):
Code:
11:22:01.951438 IP 85.27.26.66.38209 > 10.129.23.125.22: Flags [ S], seq 2750797465, win 29200, options [mss 1452,sackOK,TS val 324902832 ecr 0,nop,wscale 7], length 0
But I can't seen no incoming ssh traffic on 10.129.23.125 port 22.
 
Are you seeing this on vtnet1? That would mean it's sending the packet out. Perhaps it's blocked somewhere else on your network, somewhere between this host and the destination?
 
Are you seeing this on vtnet1? That would mean it's sending the packet out. Perhaps it's blocked somewhere else on your network, somewhere between this host and the destination?

Yes, it's a tcpdump on vtnet1.

I disabled the firewall on the destination host.

From the forwarding host, I can ssh to 10.129.23.125 port 22

Code:
# ssh 10.129.23.125 -p 22

When connecting this way, I can see packets going out on the forwarding host and coming in on the destination host.

It seem's the access is direct :

Code:
# traceroute 10.129.23.125
traceroute to 10.129.23.125 (10.129.23.125), 64 hops max, 40 byte packets
 1  10.129.23.125 (10.129.23.125)  1.535 ms  0.467 ms  0.443 ms

Does I have to enable bridging ?

Something like :

Code:
# sysctl net.link.ether.bridge.config=fxp0:0,xl0:0
# sysctl net.link.ether.bridge.ipfw=1
# sysctl net.link.ether.bridge.enable=1
 
Don't enable bridging. A bridge is used to connect two networks on layer2, basically merging your internal and external networks to a single network. Definitely not what you want.
 
I don't see anything wrong with what you have done there in your setup. Curious to see what you find out.
Maybe "I disabled the firewall on the destination host." - can you enable it and add an entry for the assignet port?
These kinds of errors I get all the time with other stuff, some things ive never found out why its not working.. So would be cool if you find the solution and paste it here.
 
Hello,

Sorry for the delay. I begun everything from zero...

Forwarding host :

Code:
root@:~ # ifconfig
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:df:33:01
    inet6 fe80::601:38ff:fedf:3301%vtnet0 prefixlen 64 tentative scopeid 0x1 
    inet 37.139.24.126 netmask 0xffffff00 broadcast 37.139.24.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:df:33:02
    inet 10.129.22.168 netmask 0xffff0000 broadcast 10.129.255.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160

/etc/pf.conf

Code:
rdr log on vtnet0 proto { tcp, udp } from any to 37.139.24.126 port 23 -> 10.129.22.244 port 22
rdr pass on vtnet0 proto icmp from any to 37.139.24.126 -> 10.129.22.244
set skip on lo0
pass out log on vtnet1 all
pass in all keep state

/etc/sysctl.conf

Code:
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet6.ip6.forwarding=1

/etc/pf.conf

Code:
sshd_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
enable_gateway="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

Final destination host :

Code:
root@:~ % ifconfig
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:e0:52:01
    inet6 fe80::601:38ff:fee0:5201%vtnet0 prefixlen 64 tentative scopeid 0x1 
    inet 95.85.3.4 netmask 0xffffff00 broadcast 95.85.3.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:e0:52:02
    inet 10.129.22.244 netmask 0xffffff00 broadcast 10.129.22.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I split the rdr and pass rules. I logged both with pflog. I was able to see that both rules matched :

Code:
root@:~ # tcpdump -n -e -r /var/log/pflog
07:05:35.366960 rule 0..16777216/0(match): rdr in on vtnet0: 89.57.26.176.34194 > 10.129.22.244.22: Flags [ S], seq 1264253441, win 29200, options [mss 1452,sackOK,TS[|tcp]>
07:05:35.367010 rule 0..16777216/0(match): pass out on vtnet1: 89.57.26.176.34194 > 10.129.22.244.22: Flags [ S], seq 1264253441, win 29200, options [mss 1452,sackOK,TS[|tcp]>

I could see the packets redirected and passing out :

Code:
root@:~ # pfctl -Psn -vv
@0 rdr log on vtnet0 inet proto tcp from any to 37.139.24.126 port = 23 -> 10.129.22.244 port 22
  [ Evaluations: 10        Packets: 10        Bytes: 584         States: 2     ]
  [ Inserted: uid 0 pid 400 State Creations: 3     ]

root@:~ # pfctl -Psr -vv
@0 pass out log on vtnet1 all flags S/SA keep state
  [ Evaluations: 10        Packets: 10        Bytes: 584         States: 2     ]
  [ Inserted: uid 0 pid 400 State Creations: 3     ]

I dumped the outgoing traffic and could see the redirected packages :

Code:
root@:~ # tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
07:22:17.169657 ARP, Reply 10.129.22.244 is-at 04:01:38:e0:52:02, length 46
07:22:17.169678 IP 89.57.26.176.35100 > 10.129.22.244.22: Flags [ S], seq 3401293392, win 29200, options [mss 1452,sackOK,TS val 731706410 ecr 0,nop,wscale 7], length 0
07:22:18.144146 IP 89.57.26.176.35100 > 10.129.22.244.22: Flags [ S], seq 3401293392, win 29200, options [mss 1452,sackOK,TS val 731706660 ecr 0,nop,wscale 7], length 0
07:22:20.152874 IP 89.57.26.176.35100 > 10.129.22.244.22: Flags [ S], seq 3401293392, win 29200, options [mss 1452,sackOK,TS val 731707162 ecr 0,nop,wscale 7], length 0
07:22:24.160756 IP 89.57.26.176.35100 > 10.129.22.244.22: Flags [ S], seq 3401293392, win 29200, options [mss 1452,sackOK,TS val 731708164 ecr 0,nop,wscale 7], length 0

But nothing arrives on the target.

Direct ssh is working :

Code:
root@:~ # tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
07:33:25.793083 IP 10.129.22.168.39389 > 10.129.22.244.22: Flags [ S], seq 4248045955, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 528779 ecr 0], length 0
07:33:25.794151 IP 10.129.22.244.22 > 10.129.22.168.39389: Flags [ S.], seq 551525078, ack 4248045956, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 559623158 ecr 528779], length 0

I used tcptraceroute to try to get some clues of what's wrong :

Code:
root@89.57.26.176 # tcptraceroute 37.139.24.126 23 
traceroute to 37.139.24.126 (37.139.24.126), 30 hops max, 60 byte packets
...
 8  37.139.24.126 (37.139.24.126)  48.847 ms  48.176 ms  48.205 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
...

It seems forwarded traffic don't find a way out to 10.129.22.244 but direct traffic well...

I think PF is working well but suspect a network problem.

Code:
root@:~ # netstat -rs
routing:
    0 bad routing redirects
    0 dynamically created routes
    0 new gateways due to redirects
    17 destinations found unreachable       
    0 uses of a wildcard route
    0 routes not in table but not freed

This is my routing table :

Code:
root@:~ # netstat -4rnd
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            37.139.24.1        UGS      vtnet0
10.129.0.0/16      link#2             U        vtnet1
10.129.22.168      link#2             UHS         lo0
37.139.24.0/24     link#1             U        vtnet0
37.139.24.126      link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0

I am completely stuck...
 
Does the machine at 10.129.22.244 have a proper gateway set? If there's no gateway it would explain why a "directly connected" host is able to connect but everything else isn't. And are you able to run tcpdump(1) there? With it you can at least make sure your packets are arriving correctly at the destination.

Don't set these in /etc/sysctl.conf:
Code:
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
Instead use these in /etc/rc.conf:
Code:
gateway_enable="YES"
ipv6_gateway_enable="YES"
It does exactly the same but nobody expects those settings in /etc/sysctl.conf.

This one's wrong:
Code:
enable_gateway="YES"
It's
Code:
gateway_enable="YES"
 
Ah. Have a really close look at vtnet1 on both hosts. Specifically the subnet mask. One has a /16 and the other /24. Possible (*), but you're setting yourself up for some tricky routing issues. Use the same subnet and mask on both hosts.

(*) It's called supernetting. But it's a royal PITA to do it correctly. As a relative newcomer to networking I advise you to not go there just yet. Stick to basic routing.
https://en.wikipedia.org/wiki/Supernetwork
 
Hello SirDice,

I remove the entries in /etc/sysctl.conf and replaced them by
Code:
gateway_enable="YES"
ipv6_gateway_enable="YES"
in /etc/rc.conf
Code:
root@:~ # sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1
root@:~ # sysctl net.inet6.ip6.forwarding
net.inet6.ip6.forwarding: 1



Ah. Have a really close look at vtnet1 on both hosts. Specifically the subnet mask. One has a /16 and the other /24. Possible (*), but you're setting yourself up for some tricky routing issues. Use the same subnet and mask on both hosts.

The different netmask was not intended. I reset the initial setup, both netmasks to /16 :
Code:
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:df:33:02
    inet 10.129.22.168 netmask 0xffff0000 broadcast 10.129.255.255
Code:
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 04:01:38:e0:52:02
    inet 10.129.22.244 netmask 0xffff0000 broadcast 10.129.255.255

And are you able to run tcpdump(1) there? With it you can at least make sure your packets are arriving correctly at the destination.

I can run tcpdump on 10.129.22.244 and see packets arriving correctly at the destination with direct connection.

Forwarding host (10.129.22.168) :
Code:
root@:~ # ssh 10.129.22.244
Code:
root@:~ # tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
07:51:59.915215 IP 10.129.22.168.24800 > 10.129.22.244.22: Flags [ S], seq 165501669, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 117902 ecr 0], length 0
07:51:59.916198 IP 10.129.22.244.22 > 10.129.22.168.24800: Flags [ S.], seq 1531336109, ack 165501670, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1528904943 ecr 117902], length 0
07:51:59.916246 IP 10.129.22.168.24800 > 10.129.22.244.22: Flags [.], ack 1, win 1040, options [nop,nop,TS val 117904 ecr 1528904943], length 0

Target host (10.129.22.244) :
Code:
root@:~ # tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
07:51:59.920228 IP 10.129.22.168.24800 > 10.129.22.244.22: Flags [ S], seq 165501669, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 117902 ecr 0], length 0
07:51:59.920327 IP 10.129.22.244.22 > 10.129.22.168.24800: Flags [ S.], seq 1531336109, ack 165501670, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1528904943 ecr 117902], length 0
07:51:59.920990 IP 10.129.22.168.24800 > 10.129.22.244.22: Flags [.], ack 1, win 1040, options [nop,nop,TS val 117904 ecr 1528904943], length 0

In case of forwarded traffic nothing arrives on the target host, "tcpdump -nni vtnet1" produces no output.



Does the machine at 10.129.22.244 have a proper gateway set?

What do you mean ?

This is my routing table on 10.129.22.244 :

Code:
Destination        Gateway            Flags      Netif Expire
default            95.85.3.1          UGS      vtnet0
10.129.0.0/16      link#2             U        vtnet1
10.129.22.244      link#2             UHS         lo0
95.85.3.0/24       link#1             U        vtnet0
95.85.3.4          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0

I am not familiar with "link#" and don't know how to interpret this output. Is it ok ?
 
In case of forwarded traffic nothing arrives on the target host, "tcpdump -nni vtnet1" produces no output.
Ok, but the packets are being transmitted by the forwarding host? If that host is transmitting but the receiving end doesn't get them there's something inbetween those two hosts that's blocking the traffic.

This is my routing table on 10.129.22.244 :
This is going to cause problems. If/When the forwarding works packets will arrive on vtnet1 but the replies will be sent out through vtnet0.

What exactly is the issue you're trying to solve?
 
Hello,

"Ok, but the packets are being transmitted by the forwarding host?"

As far as I can see the packet reach vtnet1 so they are transmitted :

My pc (89.57.26.176) :

Code:
$ ssh 37.139.24.126 -p 23

Forwarding host (37.139.24.126 - 10.129.22.168) :

Code:
# tcpdump -nni vtnet0 'port 23'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:31:56.468980 IP 89.57.26.176.36459 > 37.139.24.126.23: Flags [ S], seq 2366250510, win 29200, options [mss 1452,sackOK,TS val 2460870 ecr 0,nop,wscale 7], length 0
13:31:57.466437 IP 89.57.26.176.36459 > 37.139.24.126.23: Flags [ S], seq 2366250510, win 29200, options [mss 1452,sackOK,TS val 2461120 ecr 0,nop,wscale 7], length 0

Redirected :

Code:
# tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
13:33:32.690324 IP 89.57.26.176.36464 > 10.129.22.244.22: Flags [ S], seq 2360667904, win 29200, options [mss 1452,sackOK,TS val 2484925 ecr 0,nop,wscale 7], length 0
13:33:33.687214 IP 89.57.26.176.36464 > 10.129.22.244.22: Flags [ S], seq 2360667904, win 29200, options [mss 1452,sackOK,TS val 2485175 ecr 0,nop,wscale 7], length 0

Target host (10.129.22.244) :

Code:
# tcpdump -nni vtnet1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 65535 bytes

Nothing arrives.

"If that host is transmitting but the receiving end doesn't get them there's something in between those two hosts that's blocking the traffic."

I suspected something blocking and contacted the support of my provider (digitalocean) they ensure me they don't block anything. Port forwarding is allowed and show me a tutorial for a linux+iptables port forwarding setup.

The target host is the next hop, so there seems to be nothing between the hosts.

"This is going to cause problems. If/When the forwarding works packets will arrive on vtnet1 but the replies will be sent out through vtnet0."

No forwarded traffic reaches the target host, on any interface.

"What exactly is the issue you're trying to solve? "

I am trying to redirect ssh traffic from my pc trough to host on a private network trough a gateway.

On the forwarding host, it seems the traffic is redirect well to vtnet1 (private network) but never reach the target host. I suspect a network problem.

Peter.
 
Hello Peter,
Have you solved port forwarding? I have the same task and can not find proper solution. nat does not forward packets to external ip addresses.
 
Back
Top