Hello all,
I'm asking for help for an issue concerning the brand-new firewall I just set up. The typology is quite simple.
I use FreeBSD 10.0-RELEASE #0 with the following options:
➜ ~
➜ ~
The problem concerns file transfers with SMB2 over-TCP but seem to impact other TCP flows of other kinds but seems to occur only in the IPSec tunnel with files (or chunk of data) greater than 64KB.
IPSec tunnel is up and running, up to a 64K file no problem. But if I try with a bigger file it fails. For example, if it's a file copy, the process keep calculating and never ends. If I copy/paste a big amount of data from my clipboard to a server clipboard through RDP, the connection freezes and then returns but the data is never pasted. If it's a big SQL request to execute through SQL Manager Studio, the request seems to never arrive on the server.
The only common denominator is this 64K chunk.
I checked the MTU and the MSS in the tunnel are good. Rules seem to be ok too. The only way to make it work is to
(internal interface) and reload PF.
➜ ~
➜ ~
➜ ~
➜ ~
➜ ~
➜ ~
➜ ~
Does anyone has a single idea of what is going on ? I'm stuck.
What do I miss? TCP fine tuning (TCP buffers)? A FreeBSD, PF, ALTQ bug or misconfiguration? Maybe it's simpler.
Any help appreciated. If you need more details just ask.
Best Regards,
I'm asking for help for an issue concerning the brand-new firewall I just set up. The typology is quite simple.
Code:
vmx1 vmx0
192.168.37.2 10.38.33.2 10.38.33.1 X.X.X.X Y.Y.Y.Y 192.168.46.1
+ + + + + +
| | | | | |
| | | | | |
+-+--------------------------+-+ | | | | | | +-+--------------------------+-+
| +--------------------------+ | v v v v v v | +--------------------------+ |
| | | | | | | |
| | | | ++-----------------++ +-----------------+ ++-----------------++ | | | |
| | | | || || | +------------------| || | | | |
| | INSIDE | +-----------------| FreeBSD + PF |-----------------+ Router A | || Juniper SRX |-----------------------+ | NETWORK A | |
| | 192.168.37.0/24 | | || +ALTQ || | +------------------| || | | 192.168.46.0/24 | |
| | | | ++-----------------++ +-----------------+ ++-----------------++ | | | |
| | | | | | | |
| | | | ^ ^ | | | |
| +--------------------------+ | | | | +--------------------------+ |
+-+--------------------------+-+ | | +-+--------------------------+-+
| |
+--------------------------------------------------------+ +---------------------------------+ +----------------------------------+ +------------------------------+
| |
LAN | LAN Internet | LAN
| |
| |
+----------------------------------------------------+
IPSEC TUNNEL➜ ~
uname -a
Code:
FreeBSD testarossa 10.0-RELEASE FreeBSD 10.0-RELEASE #0: Mon Jul 21 14:09:49 CEST 2014 root@testarossa:/usr/obj/usr/src/sys/FIREWALL amd64➜ ~
cat /root/kernels/FIREWALL
Code:
include GENERIC
ident FIREWALL
device carp
device pf
device pflog
device pfsync
device crypto # IPsec depends on this
device enc
options ALTQ
options ALTQ_CBQ # Class Bases Queuing (CBQ)
options ALTQ_RED # Random Early Detection (RED)
options ALTQ_RIO # RED In/Out
options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC)
options ALTQ_PRIQ # Priority Queuing (PRIQ)
options ALTQ_NOPCC # Required for SMP build
options IPSEC
options IPSEC_DEBUG
options IPSEC_NAT_T
options IPSEC_FILTERTUNNELThe problem concerns file transfers with SMB2 over-TCP but seem to impact other TCP flows of other kinds but seems to occur only in the IPSec tunnel with files (or chunk of data) greater than 64KB.
IPSec tunnel is up and running, up to a 64K file no problem. But if I try with a bigger file it fails. For example, if it's a file copy, the process keep calculating and never ends. If I copy/paste a big amount of data from my clipboard to a server clipboard through RDP, the connection freezes and then returns but the data is never pasted. If it's a big SQL request to execute through SQL Manager Studio, the request seems to never arrive on the server.
The only common denominator is this 64K chunk.
I checked the MTU and the MSS in the tunnel are good. Rules seem to be ok too. The only way to make it work is to
Code:
set skip on vmx1➜ ~
ifconfig
Code:
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:70:d1:3f
inet 10.38.33.2 netmask 0xfffffff8 broadcast 10.38.33.7
inet6 fe80::20c:29ff:fe70:d13f%vmx0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:70:d1:49
inet6 fe80::20c:29ff:fe70:d149%vmx1 prefixlen 64 scopeid 0x2
inet 192.168.37.2 netmask 0xffffff00 broadcast 192.168.37.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:70:d1:53
inet 192.168.38.1 netmask 0xfffffffc broadcast 192.168.38.3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>➜ ~
sudo racoonctl show-sa isakmp
Code:
Destination Cookies Created
Y.Y.Y.Y.4500 f03cfeb3c1316534:1cf8debab4777de2 2014-10-13 05:04:52➜ ~
sudo cat /etc/ipsec.conf
Code:
spdadd 192.168.37.0/24 192.168.46.0/27 any -P out ipsec esp/tunnel/10.38.33.2-Y.Y.Y.Y/require;
spdadd 192.168.46.0/27 192.168.37.0/24 any -P in ipsec esp/tunnel/Y.Y.Y.Y-10.38.33.2/require;➜ ~
sudo racoonctl show-sa esp
Code:
10.38.33.2[4500] Y.Y.Y.Y[4500]
esp-udp mode=tunnel spi=4249873632(0xfd4fece0) reqid=0(0x00000000)
E: 3des-cbc d518e4d8 5329da15 bb8b3706 d5e574a2 392de16e 3bee8059
A: hmac-md5 a122f418 598912d3 62df94b6 44a17b28
seq=0x00006c0f replay=4 flags=0x00000000 state=mature
created: Oct 13 10:53:06 2014 current: Oct 13 11:27:08 2014
diff: 2042(s) hard: 3600(s) soft: 2880(s)
last: Oct 13 11:27:07 2014 hard: 0(s) soft: 0(s)
current: 3539824(bytes) hard: 0(bytes) soft: 0(bytes)
allocated: 27663 hard: 0 soft: 0
sadb_seq=3 pid=88592 refcnt=4
Y.Y.Y.Y[4500] 10.38.33.2[4500]
esp-udp mode=tunnel spi=10734330(0x00a3cafa) reqid=0(0x00000000)
E: 3des-cbc d8d7eb1a 46271766 2946cc1f de694ea6 ea94d406 c50ec435
A: hmac-md5 63034112 9c546467 6d850138 8faec77c
seq=0x00006f94 replay=4 flags=0x00000000 state=mature
created: Oct 13 10:53:06 2014 current: Oct 13 11:27:08 2014
diff: 2042(s) hard: 3600(s) soft: 2880(s)
last: Oct 13 11:27:07 2014 hard: 0(s) soft: 0(s)
current: 7576089(bytes) hard: 0(bytes) soft: 0(bytes)
allocated: 28564 hard: 0 soft: 0
sadb_seq=2 pid=88592 refcnt=1➜ ~
sudo pfctl -si
Code:
Status: Enabled for 4 days 02:17:53 Debug: Urgent
State Table Total Rate
current entries 316
searches 78109321 220.7/s
inserts 689558 1.9/s
removals 689242 1.9/s
Counters
match 1120363 3.2/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 8132 0.0/s
state-insert 62 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s➜ ~
sudo pfctl -sn
Code:
nat-anchor "ftp-proxy/*" all
nat on vmx0 inet from 192.168.37.0/24 to any -> (vmx0) round-robin
rdr-anchor "ftp-proxy/*" all
rdr pass on vmx0 inet proto tcp from any to (vmx0) port = ftp -> 127.0.0.1 port 8121
rdr pass on vmx1 inet proto tcp from 192.168.37.0/24 to ! <table> port = ftp -> 127.0.0.1 port 8021
rdr pass on vmx0 inet proto udp from any to any port = openvpn -> 192.168.36.254 port 1194
rdr on vmx0 inet proto tcp from any to (vmx0) port = 50080 -> 192.168.37.176 port 8080
rdr on vmx0 inet proto tcp from any to (vmx0) port = 56781 -> 192.168.37.172 port 81
rdr on vmx0 inet proto tcp from any to (vmx0) port = 50081 -> 192.168.37.166 port 8080➜ ~
sudo pfctl -vvs rules | grep @
Code:
@0 scrub in all no-df min-ttl 100 max-mss 1440 fragment reassemble
@0 block drop log (all) all
@1 pass quick on enc0 all flags any keep state
@2 block drop in log quick on vmx0 proto tcp from <sshguard:0> to any port = ssh label "ssh bruteforce"
@3 pass in on vmx1 inet6 from ::1 to any flags any keep state
@4 pass in on vmx1 inet from 127.0.0.1 to any flags any keep state
@5 pass in on vmx1 inet from 192.168.37.0/24 to any flags any keep state
@6 pass inet proto icmp all icmp-type echoreq keep state
@7 pass inet proto icmp all icmp-type unreach keep state
@8 pass out on vmx0 inet proto udp from any to any port 33433 >< 33626 keep state
@9 pass out proto tcp from any to any port = ssh flags S/SA keep state
@10 pass out proto tcp from any to any port = smtp flags S/SA keep state
@11 pass out proto tcp from any to any port = submission flags S/SA keep state
@12 pass out proto tcp from any to any port = domain flags S/SA keep state
@13 pass out proto tcp from any to any port = http flags S/SA keep state
@14 pass out proto tcp from any to any port = pop3 flags S/SA keep state
@15 pass out proto tcp from any to any port = auth flags S/SA keep state
@16 pass out proto tcp from any to any port = https flags S/SA keep state
@17 pass out proto tcp from any to any port = pop3s flags S/SA keep state
@18 pass out proto tcp from any to any port = imaps flags S/SA keep state
@19 pass out proto tcp from any to any port = imap flags S/SA keep state
@20 pass out proto tcp from any to any port = ntp flags S/SA keep state
@21 pass out proto tcp from any to any port = ftp flags S/SA keep state
@22 pass out proto tcp from any to any port = rdp flags S/SA keep state
@23 pass out proto tcp from any to any port = isakmp flags S/SA keep state
@24 pass out proto tcp from any to any port = sae-urn flags S/SA keep state
@25 pass out proto udp from any to any port = domain keep state
@26 pass out proto udp from any to any port = ntp keep state
@27 pass out proto udp from any to any port = isakmp keep state
@28 pass out proto udp from any to any port = sae-urn keep state
@29 pass out proto udp from any to <sip_nets:2> keep state queue sip_out
@30 pass out inet proto tcp from any to 1.2.3.4 port = 2222 flags S/SA keep state
@31 pass out inet proto tcp from any to 5.6.7.8 port = 8443 flags S/SA keep state
@32 pass out inet proto tcp from any to 9.10.11.12 port = commplex-main flags S/SA keep state
@33 pass out inet proto tcp from any to 13.14.15.16 port = commplex-link flags S/SA keep state
@34 pass in proto tcp from any to any port = ssh flags S/SA keep state
@35 pass in proto tcp from any to any port = isakmp flags S/SA keep state
@36 pass in proto tcp from any to any port = sae-urn flags S/SA keep state
@37 pass in proto udp from any to any port = isakmp keep state
@38 pass in proto udp from any to any port = sae-urn keep state
@39 pass in proto udp from <sip_nets:2> to any keep state queue sip_in
@40 pass in inet proto udp from 10.38.33.3 to (vmx0:1) port = snmp keep state
@41 pass in log proto udp from any to any port = openvpn keep state
@42 pass out log proto udp from any to any port = openvpn keep state
@43 pass in proto tcp from any to (vmx0:2) port = 50080 flags S/SA keep state
@44 pass in proto tcp from any to (vmx0:2) port = 56781 flags S/SA keep state
@45 pass in proto tcp from any to (vmx0:2) port = 50081 flags S/SA keep state
@46 pass in proto tcp from any to (vmx0:2) port = ftp flags S/SA keep state
@47 anchor "ftp-proxy/*" allDoes anyone has a single idea of what is going on ? I'm stuck.
What do I miss? TCP fine tuning (TCP buffers)? A FreeBSD, PF, ALTQ bug or misconfiguration? Maybe it's simpler.
Any help appreciated. If you need more details just ask.
Best Regards,