PF and IPSEC - problem

Hello!

I'm trying to set up IPSEC VPN on FreeBSD router with PF firewall already enabled.

Everything seems to work, hosts are accessible on both networks, BUT transmission of data fails after transmitting 128K.

Code:
[dkazarov@gw-tw ~]$ scp 2 192.168.5.5:/dev/null
2     7%  128KB 128.0KB/s   00:13 ETAWrite failed: Operation not permitted
lost connection

Disabling PF (/etc/rc.d/pf stop) solved the problem.

Code:
[dkazarov@gw-tw ~]$ /etc/rc.d/pf stop
Disabling pf.
[dkazarov@gw-tw ~]$ scp 2 192.168.5.5:/dev/null
2     100% 1827KB  65.3KB/s   00:28    
[dkazarov@gw-tw ~]$
But IMHO it is not a good idea to leave internet firewall without firewall enabled. So a bit later I found that disabling PF just on tunnel interface (set skip on gif) also solved a problem. And even completely permissive PF configuration breaks normal VPN work.

My current /etc/pf.conf
Code:
#============================>         Macros        <============================#
if_inet         = "em0"
if_lan_Office   = "em1"
pubIP           = "aa.aa.aa.130"
#============================>         Tables        <============================#
#============================>        Options        <============================#
set block-policy drop
set skip on lo0
set skip on MskPP
#============================> Traffic Normalization <============================#
#scrub in on $if_inet all fragment reassemble
#============================>        Queueing       <============================#
#============================>      Translation      <============================#
nat on $if_inet from $LAN_network to any -> $pubIP
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass on $if_lan_Office proto tcp from              any to             any port 21 -> 127.0.0.1 port 8021

# We are just setting the office and everything is allowed.
pass quick all
# Everything below does not work
# Allow any packet to go out.
pass  out all
# We have to check incoming traffic
block in  log all

When I comment "set skip on MskPP" - problem returns. With this option (set skip on MskPP) IPSEC works fine.

Is it possible to use PF with IPSEC? Why fully permissible PF disturbs IPSEC?

Other configurations:

ifconfig
Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:25:90:3c:a5:d0
        inet aa.aa.aa.130 netmask 0xffffffe0 broadcast aa.aa.aa.159
        inet6 fe80::225:90ff:fe3c:a5d0%em0 prefixlen 64 scopeid 0x2 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:25:90:3c:a5:d1
        inet 192.168.150.5 netmask 0xffffff00 broadcast 192.168.150.255
        inet6 fe80::225:90ff:fe3c:a5d1%em1 prefixlen 64 scopeid 0x3 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
MskPP: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
        tunnel inet aa.aa.aa.130 --> bb.bb.bb.123
        inet6 fe80::225:90ff:fe3c:a5d0%MskPP prefixlen 64 tentative scopeid 0x6 
        inet 192.168.150.5 --> 192.168.5.5 netmask 0xffffffff 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        options=1<ACCEPT_REV_ETHIP_VER>

Code:
[root@gw-tw ~]# netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            aa.aa.aa.129     UGS         1    45204    em0
aa.aa.aa.128/27  link#2             U           0       54    em0
aa.aa.aa.130     link#2             UHS         0        0    lo0
127.0.0.1          link#5             UH          0      390    lo0
192.168.4.0/23     MskPP              US          0    11960  MskPP
192.168.5.5        link#6             UH          0     8084  MskPP
192.168.50.0/24    MskPP              US          0      114  MskPP
192.168.150.0/24   link#3             U           0      935    em1
192.168.150.5      link#3             UHS         1        0    lo0

/etc/ipsec.conf
Code:
flush;
spdflush;

#################################################################################################################
# Main office
######################
spdadd bb.bb.bb.123 aa.aa.aa.130 ipencap -P  in ipsec esp/tunnel/bb.bb.bb.123-aa.aa.aa.130/unique;
spdadd aa.aa.aa.130 bb.bb.bb.123 ipencap -P out ipsec esp/tunnel/aa.aa.aa.130-bb.bb.bb.123/unique;
#################################################################################################################

/usr/local/etc/racoon/racoon.conf
Code:
path include "/usr/local/etc/racoon" ;
path pre_shared_key "/usr/local/etc/racoon/psk.txt" ;
padding {
        maximum_length 20;
        randomize off;
        strict_check off;
        exclusive_tail off;
}
listen {
        isakmp aa.aa.aa.130   [500];
        strict_address;
}
timer {
        counter 5;
        interval 20 sec;
        persend 1;
        phase1 30 sec;
        phase2 15 sec;
}
sainfo anonymous {
        pfs_group 2;
        lifetime time 30 sec;
        encryption_algorithm 3des;
        authentication_algorithm hmac_md5;
        compression_algorithm deflate;
}
remote bb.bb.bb.123 {
        exchange_mode main,aggressive;
        doi ipsec_doi;
        situation identity_only;
        my_identifier address aa.aa.aa.130;
        nonce_size 16;
        lifetime time 12 hour;
        initial_contact on;
        proposal_check obey;
        proposal {
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method pre_shared_key ;
                dh_group 2 ;
        }
}
 
The problem exists only when GIF tunnel is encrypted with IPSEC. When I disable IPSEC - tunnel works correctly with PF.

pass on $if_ipsec all

I've just tested this code - it does not help.
I've tried to disable nat, pass ipencap on gif
Code:
pass on MskPP proto ipencap all no state
"no state" - since tcpdump shows incoming packets on gif interfaces as ipencap.

No help. Something breaks connection after some kilobytes of data is passed:
Code:
$ scp test-garbage.bin 192.168.5.5:/dev/null
test-garbage.bin                 0%  128KB 128.0KB/s   02:39 ETAWrite failed: Operation not permitted
lost connection

Only
Code:
set skip on MskPP
solves the problem.

The problem exists on both 8.2-STABLE and 9.0-STABLE.
 
You can try clearing the dont-fragment bit and/or lower MSS, kind of:

Code:
scrub on $if_ipsec all no-df max-mss 1340
 
Code:
scrub on $if_ipsec all no-df max-mss 1340
Does not help.

New pf.conf

Code:
if_inet = "em0"
if_lan_Office = "em1"
pubIP = "aa.aa.aa.130"
LAN_network = "192.168.150.0/24"
table <self> { self }
set block-policy drop
set skip on lo0
scrub on $if_inet all fragment reassemble
scrub on MskPP all no-df max-mss 1340
nat on $if_inet from $LAN_network to any -> $pubIP
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass on $if_lan_Office proto tcp from any to any port 21 -> 127.0.0.1 port 8021
pass quick all

Also tried
Code:
scrub on MskPP all fragment reassemble

Result is the same:
Code:
# scp test-garbage.bin 192.168.5.5:/dev/null
test-garbage.bin     0%  176KB 108.5KB/s   03:07 ETAWrite failed: Operation not permitted
lost connection
# scp test-garbage.bin 192.168.5.5:/dev/null
test-garbage.bin     0%  176KB 108.5KB/s   03:07 ETAWrite failed: Operation not permitted
lost connection
 
CoTones said:
You can try lower MSS even more.
Lowered it to 800 - failure.

CoTones said:
Btw, ping doesn't go through too?

ping works:
Code:
[root ~]# ping -s 10000 -c 2 192.168.5.5
PING 192.168.5.5 (192.168.5.5): 10000 data bytes
10008 bytes from 192.168.5.5: icmp_seq=0 ttl=64 time=440.540 ms
10008 bytes from 192.168.5.5: icmp_seq=1 ttl=64 time=491.576 ms

--- 192.168.5.5 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 440.540/466.058/491.576/25.518 ms
ping, all other services, tcp connections, everything works normally if it's slow enough.
Code:
[~]$ scp -l 50 test-garbage.bin 192.168.5.5:/dev/null
test-garbage.bin        0%   96KB  77.8KB/s   04:22 ETAWrite failed: Operation not permitted
test-garbage.bin     0%   96KB  37.2KB/s - stalled -lost connection
[~]$ scp -l 40 test-garbage.bin 192.168.5.5:/dev/null
test-garbage.bin    40% 8192KB   4.7KB/s   43:55 ETA
It looks like ICMP squench packets are blocked somewhere or cause problem to PF.
 
Did you found a solution? I ran into exactly the same issue in my IPSEC Setup.

Code:
root@HostB# scp -6 test.bin user@\[HostA\]:~
test.bin          2%  176KB   1.4KB/s - stalled -

My pflog on the target Host shows the following:
Code:
00:00:00.000000 rule 0..16777216/0(match): block in on re0: (hlim 64, next-header TCP (6) payload length: 20) HostB.56168 > HostA.22: Flags [R], cksum 0xcea4 (correct), seq 25972626, win 0, length 0
00:00:08.380174 rule 0..16777216/0(match): block in on re0: (hlim 64, next-header TCP (6) payload length: 20) HostB.15861 > HostA.22: Flags [R], cksum 0xf94b (correct), seq 3907181319, win 0, length 0
00:00:08.189807 rule 0..16777216/0(match): block in on re0: (hlim 64, next-header TCP (6) payload length: 20) HostB.56240 > HostA.22: Flags [R], cksum 0x995d (correct), seq 1472147038, win 0, length 0
00:00:00.611527 rule 0..16777216/0(match): block in on re0: (hlim 64, next-header TCP (6) payload length: 20) HostB.32073 > HostA.22: Flags [R], cksum 0x0678 (correct), seq 2443371975, win 0, length 0
00:00:05.572571 rule 0..16777216/0(match): block in on re0: (flowlabel 0x26302, hlim 64, next-header TCP (6) payload length: 1371) HostB.23864 > HostA.22: Flags [.], cksum 0x3107 (correct), seq 3562838987:3562840326, ack 2542183575, win 1026, options [nop,nop,TS val 10994982 ecr 659454683], length 1339

The SSH Part of my pf.conf on both Hosts look like this:

Code:
pass in quick on $extern proto tcp from any to any port ssh flags S/SAFR keep state
pass out quick on $extern proto tcp from any to any port ssh keep state
 
I've not found a proper solution. The workaround for this problem is to add skip for all gif interfaces:
Code:
set skip on gif0
set skip on gif1
The traffic begins to flow normally at a price of lesser control.
 
I know this is an older thread but wanted to post some additional feedback. I am running into the same issue with pf filtering on a gif interface. I maintain a number of firewalls for different clients and I have noticed that this problem exists on a system that is running amd64 9.2-STABLE. The interesting thing is that I do not need to do a
Code:
set skip on gif0
on an i386 9.2-STABLE system. The following code works on the i386 version of pf:

Code:
IpsecIf="gif0"
TcpState="flags S/SA keep state"
UdpState="keep state"

pass in log on $IpsecIf inet proto tcp $TcpState
pass in log on $IpsecIf inet proto { udp, icmp } $UdpState

pass out log on $IpsecIf inet proto tcp $TcpState
pass out log on $IpsecIf inet proto { udp, icmp } $UdpState
pf does not behave the same between i386 and amd64 in regard to filtering on a gif interface.
 
Back
Top