Lost of Local SSH with PF firewall

In order to minimize damage on my production server, I have built a virtual testbed (production is FreeBSD 8.0 testbed as it says next is FreeBSD 8.3)

I am running FreeBSD 8.3 under Virtualbox on a Mac OSX box (host) on an intranet 10.0.1.x. The host has an IP of 10.0.1.118. The guest freeBSD FreeBSD server has an IP of 10.0.1.195. The network router is at 10.0.1.1 with an internet address of a.b.c.d.

I have an OpenVPN server, which appears to be functioning well with a working tunnel with VPNnet of 10.9.0.0/24. The network interface for the guest server is em1.

My issue is that with pf enabled I can ping but not ssh from the host to the guest on the intranet but I can both ping and ssh via the VPN. My current pf.conf is as follows:

Code:
tcp_pass = "{ 20 21 22 25 53 80 81 8010 110 137 138 139 443 445 465 587 993}"
udp_pass = "{ 137 138 139 465 587 1194}"

# --- SCRUB section ---
scrub in all

# --- NAT  rules -------------
nat pass on em1 from 10.9.0.0/24 to any -> 10.0.1.195
pass in quick proto {tcp,udp} to 10.0.1.195 port 1194
pass in quick from 10.9.0.0/24 to any   # I do have block in


# ------------------ FILTER RULES -------------------
# --- OUTGOING
pass out quick on em1 inet proto tcp to any port $tcp_pass
pass out quick on em1 inet proto udp to any port $udp_pass
pass out quick on em1 inet from any to any keep state
pass out quick from 10.0.1.195  to 10.0.1.0/24
# --- INCOMING
pass in quick on em1 inet proto udp to any port $udp_pass
pass in quick on em1 inet proto tcp to any port $tcp_pass
pass in quick  from 10.0.1.0/24 to 10.0.1.195

# --- pass incoming openvpn connections to the internal openvpn server ---
#pass in quick on em1 inet proto { tcp udp } from any to 10.9.0.2

# --- antispoof protection ---
antispoof quick for em1 inet

# --- default policy
#block log all

# --- end of pf rule set

Results of pfctl-sr

Code:
pass out quick on em1 inet proto udp from any to any port = netbios-ns keep state
pass out quick on em1 inet proto udp from any to any port = netbios-dgm keep state
pass out quick on em1 inet proto udp from any to any port = netbios-ssn keep state
pass out quick on em1 inet proto udp from any to any port = smtps keep state
pass out quick on em1 inet proto udp from any to any port = submission keep state
pass out quick on em1 inet proto udp from any to any port = openvpn keep state
pass out quick on em1 inet all flags S/SA keep state
pass out quick inet from 10.0.1.195 to 10.0.1.0/24 flags S/SA keep state
pass in quick on em1 inet proto udp from any to any port = netbios-ns keep state
pass in quick on em1 inet proto udp from any to any port = netbios-dgm keep state
pass in quick on em1 inet proto udp from any to any port = netbios-ssn keep state
pass in quick on em1 inet proto udp from any to any port = smtps keep state
pass in quick on em1 inet proto udp from any to any port = submission keep state
pass in quick on em1 inet proto udp from any to any port = openvpn keep state
pass in quick on em1 inet proto tcp from any to any port = ftp-data flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = ftp flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = ssh flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = smtp flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = domain flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = http flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = hosts2-ns flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = 8010 flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = pop3 flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = netbios-ns flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = netbios-dgm flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = netbios-ssn flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = https flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = microsoft-ds flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = smtps flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = submission flags S/SA keep state
pass in quick on em1 inet proto tcp from any to any port = imaps flags S/SA keep state
pass in quick inet from 10.0.1.0/24 to 10.0.1.195 flags S/SA keep state
block drop in quick on ! em1 inet from 10.0.0.0/8 to any
block drop in quick inet from 10.0.1.195 to any

I would assume my problem is shown in the last block drop.

Two questions:
  1. If this is in fact the issue, any suggestion on how to correct?
  2. In some of the guides I have read there is reference to a /etc/nat.conf. Should I have one?
 
This rule looks very suspicious:

Code:
block drop in quick on ! em1 inet from 10.0.0.0/8 to any

Post the output of ifconfig, you may have an address configured with a /8 netmask instead of the intended /24.

And since you're using quick already I would rearrange the filter rules to be in this order:

Code:
block (log) all

antispoof quick for em1 inet

<other block rules for blocking unwanted traffic, with quick>

pass out -rules with quick

pass in -rules with quick
 
I have not invoked any block rules besides the antispoof and scrub.

The ifconfig results for the FreeBSD guest is coming in on em2:

Code:
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether 08:00:27:c0:d2:10
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether 08:00:27:8a:ff:f7
	inet 10.0.1.195 netmask 0xff000000 broadcast 10.255.255.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet 10.9.0.1 --> 10.9.0.2 netmask 0xffffffff

ifconfig results for the Mac OSX host (Virtualbox) is:

Code:
o0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether e8:06:88:cb:d1:04 
	inet6 fe80::ea06:88ff:fecb:d104%en0 prefixlen 64 scopeid 0x4 
	inet 10.0.1.100 netmask 0xffffff00 broadcast 10.0.1.255
	media: autoselect (100baseTX <full-duplex,flow-control>)
	status: active
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	ether e8:06:88:cb:bc:05 
	inet6 fe80::ea06:88ff:fecb:bc05%en1 prefixlen 64 scopeid 0x5 
	inet 10.0.1.118 netmask 0xffffff00 broadcast 10.0.1.255
	media: autoselect (100baseTX <full-duplex,flow-control>)
	status: active
en2: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:26:08:fd:8d:2b 
	inet6 fe80::226:8ff:fefd:8d2b%en2 prefixlen 64 scopeid 0x6 
	inet 10.0.1.109 netmask 0xffffff00 broadcast 10.0.1.255
	media: autoselect
	status: active
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
	lladdr 78:ca:39:ff:fe:10:f3:78 
	media: autoselect <full-duplex>
	status: inactive
vnic0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:1c:42:00:00:08 
	inet 10.211.55.2 netmask 0xffffff00 broadcast 10.211.55.255
	media: autoselect
	status: active
vnic1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:1c:42:00:00:09 
	inet 10.37.129.2 netmask 0xffffff00 broadcast 10.37.129.255
	media: autoselect
	status: active
tun0: flags=8851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	inet 10.9.0.6 --> 10.9.0.5 netmask
 
On the FreeBSD guest the netmask is wrong for em1. Make sure you write the address for it in rc.conf(5) this way:

Code:
ifconfig_em0="inet 10.0.1.195/24"

Without the /24 the netmask will be assumed to be /8 which is wrong for your case.
 
hmmm this is interesting

Your comment on the x.x.x.x/8 sort of had me thinking I had seen that somewhere. So I pulled a netstat -nr on the guest server and I got this in the Routing Tables:

Code:
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.0.1.1           UGS         0    12279    em1
10.0.0.0/8         link#3             U           0    12272    em1
10.0.1.195         link#3             UHS         0        0    lo0
10.9.0.0/24        10.9.0.2           UGS         0    12185   tun0
10.9.0.1           link#6             UHS         0        0    lo0
10.9.0.2           link#6             UH          0        0   tun0
127.0.0.1          link#4             UH          0        0    lo0

I have to admit I have no idea where that 10.0.0.0.8 is coming from.
 
Do a test where you change the VPN tunnel addresses to for example 172.16.9.0/24 (I just took the 9 from the addresses you used). Then there shouldn't be a conflict between the external address of the VPN server and the addresses used by the tunnel.
 
Essentially I have done both of your suggestions: corrected the rc.conf to clean up the ifconfig, and changed the VPN tunnel to 172.168.9.0/24. The situation has improved somewhat. I can now ping and ssh to the guest at 10.0.1.195 until I fire up the VPN. After that point I can longer ssh to the 10.0.1.195 address, although it responds to ping even if I close the VPN. Is this indicative of a routing issue?

The host routing table before VPN and before touching the guest at 10.0.1.195 is as follows:

Code:
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            10.0.1.1           UGSc           41        0     en0
default            10.0.1.1           UGScI           0        0     en1
default            10.0.1.1           UGScI           0        0     en2
10.0.1/24          link#4             UCS             5        0     en0
10.0.1/24          link#5             UCSI            2        0     en1
10.0.1/24          link#6             UCSI            3        0     en2
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI          42       52     en0   1192
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           1        2     en1   1165
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           0        2     en2    991
10.0.1.100         127.0.0.1          UHS             0       37     lo0
10.0.1.108         e0:b9:ba:7:79:85   UHLWI           1       38     en0    704
10.0.1.109         127.0.0.1          UHS             0       28     lo0
10.0.1.110         90:84:d:f4:e3:5e   UHLWI           0       65     en0    751
10.0.1.118         127.0.0.1          UHS             0        1     lo0
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           2    10456     en0   1186
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           0       52     en2   1199
10.0.1.255         link#4             UHLWbI          1      207     en0
10.0.1.255         link#5             UHLWbI          1       11     en1
10.0.1.255         link#6             UHLWbI          2       84     en2
10.37.129/24       link#9             UC              2        0   vnic1
10.37.129.2        0:1c:42:0:0:9      UHLWI           0        1     lo0
10.37.129.255      link#9             UHLWbI          4      102   vnic1
10.211.55/24       link#8             UC              2        0   vnic0
10.211.55.2        0:1c:42:0:0:8      UHLWI           0        1     lo0
10.211.55.255      link#8             UHLWbI          2      102   vnic0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              8     1985     lo0
169.254            link#4             UCS             0        0     en0

After I ssh to the 10.0.1.195 my routing table is:

Code:
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            10.0.1.1           UGSc           41        0     en0
default            10.0.1.1           UGScI           0        0     en1
default            10.0.1.1           UGScI           0        0     en2
10.0.1/24          link#4             UCS             6        0     en0
10.0.1/24          link#5             UCSI            3        0     en1
10.0.1/24          link#6             UCSI            4        0     en2
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI          42       54     en0   1192
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           1        4     en1   1089
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           0        2     en2    936
10.0.1.100         127.0.0.1          UHS             1      124     lo0
10.0.1.108         e0:b9:ba:7:79:85   UHLWI           0       38     en0    266
10.0.1.109         127.0.0.1          UHS             1      109     lo0
10.0.1.110         90:84:d:f4:e3:5e   UHLWI           0       65     en0    313
10.0.1.118         127.0.0.1          UHS             0        1     lo0
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           1    12575     en0   1164
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           0      130     en2   1164
10.0.1.195         8:0:27:c0:d2:10    UHLWI           1       49     en0   1138
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en1   1138
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en2   1138
10.0.1.255         link#4             UHLWbI          1      349     en0
10.0.1.255         link#5             UHLWbI          1       15     en1
10.0.1.255         link#6             UHLWbI          2       88     en2
10.37.129/24       link#9             UC              2        0   vnic1
10.37.129.2        0:1c:42:0:0:9      UHLWI           0        1     lo0
10.37.129.255      link#9             UHLWbI          4      151   vnic1
10.211.55/24       link#8             UC              2        0   vnic0
10.211.55.2        0:1c:42:0:0:8      UHLWI           0        1     lo0
10.211.55.255      link#8             UHLWbI          2      151   vnic0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              8     3737     lo0
169.254            link#4             UCS             0        0     en0

With the ssh up and the VPN open it is as follows:

Code:
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0/1                172.168.9.5        UGSc            3        0    tun0
default            10.0.1.1           UGSc           11        0     en0
default            10.0.1.1           UGScI           0        0     en1
default            10.0.1.1           UGScI           0        0     en2
10.0.1/24          link#4             UCS             5        0     en0
10.0.1/24          link#5             UCSI            3        0     en1
10.0.1/24          link#6             UCSI            4        0     en2
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           3       55     en0   1200
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           1        5     en1   1110
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           0        2     en2   1104
10.0.1.100         127.0.0.1          UHS             1      153     lo0
10.0.1.108         e0:b9:ba:7:79:85   UHLWI           2       71     en0    106
10.0.1.109         127.0.0.1          UHS             0      136     lo0
10.0.1.110         90:84:d:f4:e3:5e   UHLWI           0       76     en0    153
10.0.1.118         127.0.0.1          UHS             0        1     lo0
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           2    13376     en0   1160
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           0      156     en2   1160
10.0.1.195/32      10.0.1.1           UGSc            1        0     en0
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en1    978
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en2    978
10.0.1.255         link#4             UHLWbI          1      396     en0
10.0.1.255         link#5             UHLWbI          1       15     en1
10.0.1.255         link#6             UHLWbI          2       88     en2
10.37.129/24       link#9             UC              2        0   vnic1
10.37.129.2        0:1c:42:0:0:9      UHLWI           0        1     lo0
10.37.129.255      link#9             UHLWbI          4      166   vnic1
10.211.55/24       link#8             UC              2        0   vnic0
10.211.55.2        0:1c:42:0:0:8      UHLWI           0        1     lo0
10.211.55.255      link#8             UHLWbI          2      166   vnic0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH             10     4410     lo0
128.0/1            172.168.9.5        UGSc            3        0    tun0
169.254            link#4             UCS             0        0     en0
172.168.9.1/32     172.168.9.5        UGSc            0        0    tun0
172.168.9.5        172.168.9.6        UH              9        0    tun

After I close the VPN:

Code:
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            10.0.1.1           UGSc           34        0     en0
default            10.0.1.1           UGScI           0        0     en1
default            10.0.1.1           UGScI           0        0     en2
10.0.1/24          link#4             UCS             7        0     en0
10.0.1/24          link#5             UCSI            4        0     en1
10.0.1/24          link#6             UCSI            4        0     en2
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI          25       56     en0   1190
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           1        6     en1   1077
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           0        2     en2    890
10.0.1.50          0:14:51:72:bb:fc   UHLWI           0        0     en0   1085
10.0.1.50          0:14:51:72:bb:fc   UHLWI           0        0     en1   1085
10.0.1.100         127.0.0.1          UHS             1      182     lo0
10.0.1.108         e0:b9:ba:7:79:85   UHLWI           1       18     en0   1088
10.0.1.109         127.0.0.1          UHS             1      163     lo0
10.0.1.110         90:84:d:f4:e3:5e   UHLWI           0       11     en0   1135
10.0.1.118         127.0.0.1          UHS             0        1     lo0
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           1      786     en0   1154
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           0      208     en2   1154
10.0.1.195/32      10.0.1.1           UGSc            1        0     en0
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en1    764
10.0.1.195         8:0:27:c0:d2:10    UHLWI           0        0     en2    764
10.0.1.255         ff:ff:ff:ff:ff:ff  UHLWbI          0        3     en0
10.0.1.255         link#5             UHLWbI          1       17     en1
10.0.1.255         link#6             UHLWbI          2       90     en2
10.37.129/24       link#9             UC              5        0   vnic1
10.37.129.2        0:1c:42:0:0:9      UHLWI           0        1     lo0
10.37.129.255      ff:ff:ff:ff:ff:ff  UHLWbI          1        1   vnic1
10.211.55/24       link#8             UC              4        0   vnic0
10.211.55.2        0:1c:42:0:0:8      UHLWI           0        1     lo0
10.211.55.255      ff:ff:ff:ff:ff:ff  UHLWbI          0        1   vnic0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              8     5289     lo0
169.254            link#4             UCS             0        0     en0

That routing table appears to be the same as before the VPN, so it would appear that the client cleared its routes, but I can only ping and not ssh into the guest at 10.0.1.195.
 
Is the virtual machine configured with bridged adapter or NAT? I'm assuming bridged but you haven't specified that. Why do you have so many different en adapters active at the same time on the OS X side? And if I read this right they all have the same IP address 10.0.1.1, that has to do something with the problems you're having.
 
Sorry for my oversight. I am bridging the Virtualbox to the em0 interface.[/B] em0 (10.0.1.100) and em1 (10.0.1.109) are ethernet cards, em2 is a wireless card with IP 10.0.1.108. To simplify things I took them off line.

My setup is essentially:

Code:
Internet --> Router --> OS X Host   --------> TestBed --->VPN
a.b.c.d ----> 10.0.1.1-->10.0.1.100 --->10.0.1.195----->172.168.9.0/32

I rebooted after disabling the extra interfaces and situation remains where I can ssh to the guest server until I activate the VPN. Afterwards no joy.

I am not sure my situation is really a show stopper in I don't access my production server through an intranet only the internet.

Code:
Intenet ------>  Production FreeBSD 8.0 ----->VPN
 
Just to clear what is meant by "activate the VPN". Is it when you start the openvpn service you lose ssh(1) connectivity or when you make an OpenVPN connection from a client machine?

Sorry I have to be this pedantic but all the previous information hasn't been clear and concise enough to get the important details down.
 
No problem. I am sorry about the clarity.

My use of "Activate the VPN" refers to when I establish a VPN tunnel from the client to the server. The OpenVPN server software is running all the time as it starts with boot. Upon establishing a connection, an existing ssh connection to the 10.0.1.195 address stays in place, but I am unable to establish any additional ssh connections to the 10.0.1.195. I hope that is clearer.
 
Post the following:
  • The OpenVPN server config file openvpn.conf.
  • The client configuration file.
  • Client log file that shows one successful connection.
  • The OpenVPN server log entries from the same connection.
 
Very little is working as it should now, and, as you said, you don't have much time, I will therefore drop the issue. Thanks again.
 
Back
Top