Solved From WAN connect to daemon (Inside jail) via pf rdr does not work suddely

Hi everyone.

My FreeBSD server (domain name: epopen.com) worked fine at ADSL(http/ssh/name/ftp @ jail inside), but work fail after moved to new house and use cable ISP....
Can help me for debug? Plesae.

Server description:
HW:
..fxp0: for internal LAN, fixed IP:192.168.0.254
..fxp1: for WAN, IP: DHCP, current IP: 123.194.237.215
SW:
..FreeBSD 9.1 i386 stable with some daemon...
..Daemon1: SSH server @ port 2222 in host.
..Daemon2: SSH server @ IP:10.0.0.4, port:22 in jail1.
..Daemon2: BIND9 server @ IP:10.0.0.1, port:53 in jail2.
..1.All of jail bind at lo1 IP:10.0.0.254, netmask:255.255.255.0, create by cloned_interfaces="lo1"
..2.For debug, remove addition http/ftp etc...

Test machine: windows vista
..1.ethernet connect to FreeBSD server's fxp1, fixed IP:192.168.0.1
..2.Wireless to internet.

FreeBSD server's /etc/rc.conf
Code:
kern_securelevel="2"
kern_securelevel_enable="YES"
pf_enable="YES"                                 # Set to YES to enable packet filter (pf)
pflog_enable="YES"
ifconfig_fxp1="DHCP polling"
ifconfig_fxp0="inet 192.168.0.254 netmask 255.255.255.0 polling"

cloned_interfaces="lo1"
ifconfig_lo1="inet 10.0.0.254 netmask 255.255.255.0"    # Jail host

jail_enable="YES"
jail_list="sqld sshd"                   # Space separated list of names of jails
jail_interface="lo1"                                                    # Jail's NIC interface
jail_set_hostname_allow="NO"                                    # Allow root user in a jail to change its hostname
jail_mount_enable="YES"                                                 # mount/umount jail's fs
jail_exec_start="/bin/sh /etc/rc.start"                 # command to execute in jail for starting
jail_exec_stop="/bin/sh /etc/rc.stop"                   # command to execute in jail for stoping
jail_devfs_enable="YES"                                                 # Devfs

jail_named_rootdir="/usr/jail/named"                    # Jail's root directory
jail_named_fstab="/etc/fstab.named"                             # fstab(5) for mount/umount
jail_named_hostname="dns.epopen.com"                    # Jail's hostname
jail_named_ip="10.0.0.1"                                                # Jail's IP number

jail_sshd_rootdir="/usr/jail/sshd"                              # Jail's root directory
jail_sshd_fstab="/etc/fstab.sshd"                               # fstab(5) for mount/umount
jail_sshd_hostname="ssh.epopen.com"                             # Jail's hostname
jail_sshd_ip="10.0.0.4"                                                 # Jail's IP number

Before, I used ADSL connection (ppp, interface=tun0) workable with under /etc/pf.conf
Code:
ext_if="tun0"
int_if="fxp0"

jail_named_ip="10.0.0.1"
jail_sshd_ip="10.0.0.4"
jail_named_port="domain"
jail_sshd_port="ssh"
jail_ALL_port_tcp="{" $jail_named_port "}"
jail_ALL_port_udp="{" $jail_named_port "}"

priv_net_RFC1918 = " 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 "
priv_net_DUSA = " 0.0.0.0/8, 127.0.0.1/8, 169.254.0.0/16, 192.0.2.0/24, 224.0.0.0/4, 240.0.0.0/4 "

ext_if_upstream_speed="5120Kb"

table <priv_net> const { $priv_net_RFC1918, $priv_net_DUSA }
table <jail_ip_tcp> const { $jail_named_ip }
table <jail_ip_udp> const { $jail_named_ip }
table <bruteforce> persist

set limit { states 10000, frags 5000 }
set loginterface $ext_if
set skip on lo0
set skip on lo1
set optimization aggressive
set block-policy drop
set state-policy if-bound
set require-order yes

scrub in all fragment reassemble

altq on $ext_if hfsc bandwidth $ext_if_upstream_speed queue { ext_std, ext_inet_serv, ext_inet_sys }

queue ext_std bandwidth 5% hfsc(default)
queue ext_inet_serv bandwidth 80% priority 6 hfsc(linkshare 60%, red) { ext_http, ext_ftp }
        queue ext_ftp bandwidth 20% priority 6 hfsc(linkshare 10%, red)
        queue ext_http bandwidth 80% priority 7 hfsc(linkshare 60%, red)
queue ext_inet_sys bandwidth 15% priority 7 hfsc(realtime 15%) { ext_ssh, ext_domain, ext_tcpack }
        queue ext_ssh bandwidth 20% priority 5 hfsc(realtime 20%, linkshare 10%)
        queue ext_domain bandwidth 20% priority 6 hfsc(realtime 20%, linkshare 10%)
        queue ext_tcpack bandwidth 60% priority 7 hfsc(realtime 60%)

rdr on $ext_if inet proto { udp, tcp } from any to ($ext_if) port { $jail_named_port } -> $jail_named_ip
rdr on $ext_if inet proto { tcp } from any to ($ext_if) port { $jail_sshd_port } -> $jail_sshd_ip

block all

block in quick on $ext_if from { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> } to any
block out quick on $ext_if from any to { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> }

pass quick on { $ext_if } proto pfsync

# UDP
pass out on $ext_if proto udp all keep state queue ext_std
pass out quick on $ext_if proto udp from any to any port domain keep state queue ext_domain

pass in quick on $ext_if proto udp from any to <jail_ip_udp> port $jail_ALL_port_udp keep state

# TCP
pass out quick on $ext_if proto tcp all flags S/SA modulate state queue (ext_std, ext_tcpack)  # TCP ACK Highest priority
pass out quick on $ext_if proto tcp from any to any port { $jail_sshd_port } flags S/SA modulate state queue ext_ssh
pass out quick on $ext_if proto tcp from any to any port { $jail_named_port } flags S/SA modulate state queue ext_domain

# pass in certain TCP connections and keep state (SSH, SMTP, DNS, IDENT)
pass in quick on $ext_if proto tcp from any to <jail_ip_tcp> port $jail_ALL_port_tcp flags S/SAFR synproxy state

# SSH rule
pass in quick on $ext_if proto tcp from any to $jail_sshd_ip port { $jail_sshd_port } flags S/SAFR synproxy state (max-src-conn-rate 3/30, overload <bruteforce> flush global)
pass in quick proto tcp from any to any port { 2222 } flags S/SAFR synproxy state (max-src-conn-rate 3/30, overload <bruteforce> flush global)
After, I moved to new house and use cable ISP, so I modified
Code:
ext_if="fxp1"

PF final rule result:
Code:
# pfctl -sr
scrub in all fragment reassemble
block drop all
block drop in quick on fxp1 from <priv_net> to any
block drop in quick on fxp1 from <realtimeBlockList> to any
block drop in quick on fxp1 from <bruteforce> to any
block drop in quick on fxp1 from <blockList-File> to any
block drop in quick on fxp1 from <emerging-threats> to any
block drop in quick on fxp1 from <jail_blockIP-Pool> to any
block drop out quick on fxp1 from any to <priv_net>
block drop out quick on fxp1 from any to <realtimeBlockList>
block drop out quick on fxp1 from any to <bruteforce>
block drop out quick on fxp1 from any to <blockList-File>
block drop out quick on fxp1 from any to <emerging-threats>
block drop out quick on fxp1 from any to <jail_blockIP-Pool>
pass quick on fxp1 proto pfsync all keep state (if-bound)
pass out on fxp1 proto udp all keep state (if-bound) queue ext_std
pass out quick on fxp1 proto udp from any to any port = domain keep state (if-bound) queue ext_domain
pass in quick on fxp1 proto udp from any to <jail_ip_udp> port = domain keep state (if-bound)
pass out quick on fxp1 proto tcp all flags S/SA modulate state (if-bound) queue(ext_std, ext_tcpack)
pass out quick on fxp1 proto tcp from any to any port = ssh flags S/SA modulate state (if-bound) queue ext_ssh
pass out quick on fxp1 proto tcp from any to any port = domain flags S/SA modulate state (if-bound) queue ext_domain
pass in quick on fxp1 proto tcp from any to <jail_ip_tcp> port = domain flags S/FSRA synproxy state (if-bound)
pass in quick on fxp1 inet proto tcp from any to 10.0.0.4 port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn-rate 3/30, overload <bruteforce> flush global, if-bound, src.track 30)
pass in quick proto tcp from any to any port = 2222 flags S/FSRA synproxy state (source-track rule, max-src-conn-rate 3/30, overload <bruteforce> flush global, if-bound, src.track 30)

And Ifconfig ...
Code:
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2049<RXCSUM,VLAN_MTU,POLLING,WOL_MAGIC>
        ether 00:30:64:02:8a:48
        inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::230:64ff:fe02:8a48%fxp0 prefixlen 64 scopeid 0x2
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2049<RXCSUM,VLAN_MTU,POLLING,WOL_MAGIC>
        ether 00:30:64:02:8a:49
        inet 123.194.237.215 netmask 0xfffffc00 broadcast 255.255.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        syncpeer: 0.0.0.0 maxupd: 128
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.0.0.254 netmask 0xffffff00
        inet 10.0.0.5 netmask 0xffffffff
        inet 10.0.0.4 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
But can't connect to my name & SSH server from WAN(ext_if) via pf redirect

NEXT thread continue...
 
Continue (2/3)...

Scenario 1: Jail domain name server query from internet WAN (fxp1)
Windows PC: Query domain name via internet WAN to FreeBSD Server (123.194.237.215)
Code:
C:\>nslookup
default:  dns.seed.net.tw
Address:  139.175.55.244

> epopen.com
Server:  dns.seed.net.tw
Address:  139.175.55.244

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** dns.seed.net.tw time out
FreeBSD's tcpdump result
Code:
tcpdump -vvi fxp1 'udp port 53'
tcpdump: listening on fxp1, link-type EN10MB (Ethernet), capture size 65535 bytes
04:21:28.245038 IP (tos 0x0, ttl 59, id 57558, offset 0, flags [none], proto UDP (17), length 56)
    ns6.seed.net.tw.17198 > 123-194-237-215.dynamic.kbronet.com.tw.domain: [udp sum ok] 32229 A? epopen.com. (28)
04:21:28.742358 IP (tos 0x0, ttl 64, id 14306, offset 0, flags [none], proto UDP (17), length 74)
    123-194-237-215.dynamic.kbronet.com.tw.63578 > dns.seed.net.tw.domain: [udp sum ok] 46059+ PTR? 215.237.194.123.in-addr.arpa. (46)
04:21:28.751870 IP (tos 0x0, ttl 59, id 14157, offset 0, flags [none], proto UDP (17), length 126)
    dns.seed.net.tw.domain > 123-194-237-215.dynamic.kbronet.com.tw.63578: [udp sum ok] 46059 q: PTR? 215.237.194.123.in-addr.arpa. 1/0/0 215.237.194.123.in-addr.arpa. [23h58m55s] PTR 123-194-237-215.dynamic.kbronet.com.tw. (98)
04:21:28.752908 IP (tos 0x0, ttl 64, id 2771, offset 0, flags [none], proto UDP (17), length 73)
    123-194-237-215.dynamic.kbronet.com.tw.25858 > dns.seed.net.tw.domain: [udp sum ok] 46060+ PTR? 235.55.175.139.in-addr.arpa. (45)
04:21:28.761874 IP (tos 0x0, ttl 59, id 14158, offset 0, flags [none], proto UDP (17), length 102)
    dns.seed.net.tw.domain > 123-194-237-215.dynamic.kbronet.com.tw.25858: [udp sum ok] 46060* q: PTR? 235.55.175.139.in-addr.arpa. 1/0/0 235.55.175.139.in-addr.arpa. [1d] PTR ns6.seed.net.tw. (74)
04:21:29.762751 IP (tos 0x0, ttl 64, id 39419, offset 0, flags [none], proto UDP (17), length 73)
    123-194-237-215.dynamic.kbronet.com.tw.16049 > dns.seed.net.tw.domain: [udp sum ok] 46061+ PTR? 244.55.175.139.in-addr.arpa. (45)
04:21:29.772039 IP (tos 0x0, ttl 59, id 14159, offset 0, flags [none], proto UDP (17), length 102)
    dns.seed.net.tw.domain > 123-194-237-215.dynamic.kbronet.com.tw.16049: [udp sum ok] 46061* q: PTR? 244.55.175.139.in-addr.arpa. 1/0/0 244.55.175.139.in-addr.arpa. [1d] PTR dns.seed.net.tw. (74)
04:21:30.248355 IP (tos 0x0, ttl 59, id 57559, offset 0, flags [none], proto UDP (17), length 56)
    ns6.seed.net.tw.24592 > 123-194-237-215.dynamic.kbronet.com.tw.domain: [udp sum ok] 27344 AAAA? epopen.com. (28)
Here, named doesn't reply query result...
Named log nothing...
PF state
Code:
# pfctl -ss
all tcp 192.168.0.254:2222 <- 192.168.0.1:58470       ESTABLISHED:ESTABLISHED
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 157.56.96.9:30092       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:6357       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:56763       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:45499       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:43889       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:7640       NO_TRAFFIC:SINGLE
fxp1 udp 10.0.0.1:53 (123.194.237.215:53) <- 139.175.55.235:8453       NO_TRAFFIC:SINGLE
Here state: Confuse NO_TRAFFIC:SINGLE, I don't why..

Scenario2: Jail ssh server connect from internet WAN (fxp1)
Windows PC: PuTTY connection via internet WAN to FreeBSD Server (123.194.237.215)
FreeBSD's tcpdump result
Code:
# tcpdump -vvi fxp1 'tcp port 22'
tcpdump: listening on fxp1, link-type EN10MB (Ethernet), capture size 65535 bytes
03:56:38.946077 IP (tos 0x0, ttl 127, id 7864, offset 0, flags [DF], proto TCP (6), length 48)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [S], cksum 0x43d8 (correct), seq 2337856636, win 8192, options [mss 1460,nop,nop,sackOK], length 0
03:56:38.946233 IP (tos 0x10, ttl 64, id 60126, offset 0, flags [DF], proto TCP (6), length 44)
    123-194-237-215.dynamic.kbronet.com.tw.ssh > 123-194-233-71.dynamic.kbronet.com.tw.59575: Flags [S.], cksum 0x6abe (correct), seq 45681495, ack 2337856637, win 0, options [mss 1460], length 0
03:56:38.964087 IP (tos 0x0, ttl 127, id 7865, offset 0, flags [DF], proto TCP (6), length 40)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [.], cksum 0x3e0b (correct), seq 1, ack 1, win 17520, length 0
03:56:38.964217 IP (tos 0x10, ttl 64, id 24921, offset 0, flags [DF], proto TCP (6), length 40)
    123-194-237-215.dynamic.kbronet.com.tw.ssh > 123-194-233-71.dynamic.kbronet.com.tw.59575: Flags [.], cksum 0x827b (correct), seq 1, ack 1, win 65535, length 0
03:56:39.188541 IP (tos 0x0, ttl 64, id 14301, offset 0, flags [DF], proto TCP (6), length 99)
    123-194-237-215.dynamic.kbronet.com.tw.ssh > 123-194-233-71.dynamic.kbronet.com.tw.59575: Flags [P.], cksum 0xd712 (correct), seq 1:60, ack 1, win 65535, length 59
03:56:39.209781 IP (tos 0x0, ttl 127, id 7866, offset 0, flags [DF], proto TCP (6), length 68)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0x48da (correct), seq 1:29, ack 60, win 17461, length 28
03:56:39.210750 IP (tos 0x0, ttl 127, id 7867, offset 0, flags [DF], proto TCP (6), length 552)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0xf6c9 (correct), seq 29:541, ack 60, win 17461, length 512
03:56:39.210774 IP (tos 0x0, ttl 127, id 7868, offset 0, flags [DF], proto TCP (6), length 168)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0xfdff (correct), seq 541:669, ack 60, win 17461, length 128
03:56:39.488935 IP (tos 0x0, ttl 127, id 7872, offset 0, flags [DF], proto TCP (6), length 708)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0xc3d5 (correct), seq 1:669, ack 60, win 17461, length 668
03:56:40.099346 IP (tos 0x0, ttl 127, id 7873, offset 0, flags [DF], proto TCP (6), length 708)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0xc3d5 (correct), seq 1:669, ack 60, win 17461, length 668
03:56:41.299125 IP (tos 0x0, ttl 127, id 7876, offset 0, flags [none], proto TCP (6), length 576)
    123-194-233-71.dynamic.kbronet.com.tw.59575 > 123-194-237-215.dynamic.kbronet.com.tw.ssh: Flags [P.], cksum 0xe368 (correct), seq 1:537, ack 60, win 17461, length 536
Here ssh server can't reply any data.
PuTTY connect fail.
PF state
Code:
# pfctl -ss
fxp1 tcp 10.0.0.4:22 (123.194.237.215:22) <- 123.194.233.71:59577       ESTABLISHED:ESTABLISHED
Here state: Confuse ESTABLISHED:ESTABLISHED.

Removed queue & altq not work also.

Next thread continue..
 
Continue (3/3)

For test ssh connect to sshd jail via $int_if(internal LAN), I added under
Code:
rdr on $int_if inet proto { tcp } from any to ($int_if) port { $jail_sshd_port } -> $jail_sshd_ip
pass out quick on $int_if proto tcp all flags S/SA modulate state queue (ext_std, ext_tcpack)  # TCP ACK Highest priority
pass out quick on $int_if proto tcp from any to any port { $jail_sshd_port } flags S/SA modulate state queue ext_ssh
pass in quick on $int_if proto tcp from any to $jail_sshd_ip port { $jail_sshd_port } synproxy state

Scenario3: Jail ssh server connect from LAN (fxp0)
Windows PC: PuTTY connection LAN to FreeBSD Server (192.168.0.254)
FreeBSD's tcpdump result
Code:
# tcpdump -vvi fxp0 'tcp port 22'
tcpdump: listening on fxp0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:09:57.739388 IP (tos 0x0, ttl 128, id 18904, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [S], cksum 0xaaca (correct), seq 3793901293, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
04:09:57.739553 IP (tos 0x10, ttl 64, id 25989, offset 0, flags [DF], proto TCP (6), length 44)
    192.168.0.254.ssh > 192.168.0.1.59605: Flags [S.], cksum 0x75d4 (correct), seq 2557797760, ack 3793901294, win 0, options [mss 1460], length 0
04:09:57.740377 IP (tos 0x0, ttl 128, id 18905, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [.], cksum 0x92a0 (correct), seq 1, ack 1, win 64240, length 0
04:09:57.740501 IP (tos 0x10, ttl 64, id 63257, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.0.254.ssh > 192.168.0.1.59605: Flags [.], cksum 0x8d91 (correct), seq 1, ack 1, win 65535, length 0
04:09:57.770768 IP (tos 0x0, ttl 64, id 20357, offset 0, flags [DF], proto TCP (6), length 99)
    192.168.0.254.ssh > 192.168.0.1.59605: Flags [P.], cksum 0xe228 (correct), seq 1:60, ack 1, win 65535, length 59
04:09:57.877471 IP (tos 0x0, ttl 128, id 18906, offset 0, flags [DF], proto TCP (6), length 68)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x9d6f (correct), seq 1:29, ack 60, win 64181, length 28
04:09:57.877519 IP (tos 0x0, ttl 128, id 18907, offset 0, flags [DF], proto TCP (6), length 552)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x4706 (correct), seq 29:541, ack 60, win 64181, length 512
04:09:57.877540 IP (tos 0x0, ttl 128, id 18908, offset 0, flags [DF], proto TCP (6), length 168)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x8b49 (correct), seq 541:669, ack 60, win 64181, length 128
04:09:58.172661 IP (tos 0x0, ttl 128, id 18913, offset 0, flags [DF], proto TCP (6), length 708)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x4cc6 (correct), seq 1:669, ack 60, win 64181, length 668
04:09:58.781573 IP (tos 0x0, ttl 128, id 18915, offset 0, flags [DF], proto TCP (6), length 708)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x4cc6 (correct), seq 1:669, ack 60, win 64181, length 668
04:09:59.982852 IP (tos 0x0, ttl 128, id 18917, offset 0, flags [none], proto TCP (6), length 576)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x33a5 (correct), seq 1:537, ack 60, win 64181, length 536
04:10:00.771018 IP (tos 0x0, ttl 64, id 30812, offset 0, flags [DF], proto TCP (6), length 99)
    192.168.0.254.ssh > 192.168.0.1.59605: Flags [P.], cksum 0xe228 (correct), seq 1:60, ack 1, win 65535, length 59
04:10:00.771879 IP (tos 0x0, ttl 128, id 18919, offset 0, flags [none], proto TCP (6), length 40)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [.], cksum 0x9088 (correct), seq 537, ack 60, win 64181, length 0
04:10:01.184137 IP (tos 0x0, ttl 128, id 18922, offset 0, flags [none], proto TCP (6), length 576)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x33a5 (correct), seq 1:537, ack 60, win 64181, length 536
04:10:02.384938 IP (tos 0x0, ttl 128, id 18929, offset 0, flags [DF], proto TCP (6), length 708)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x4cc6 (correct), seq 1:669, ack 60, win 64181, length 668
04:10:04.787518 IP (tos 0x0, ttl 128, id 18933, offset 0, flags [DF], proto TCP (6), length 708)
    192.168.0.1.59605 > 192.168.0.254.ssh: Flags [P.], cksum 0x4cc6 (correct), seq 1:669, ack 60, win 64181, length 668
Here ssh server can't reply any data.
PuTTY connect FAIL also.
PF state
Code:
# pfctl -ss
fxp0 tcp 10.0.0.4:22 (192.168.0.254:22) <- 192.168.0.1:59605       ESTABLISHED:ESTABLISHED
Here state: Confuse ESTABLISHED:ESTABLISHED.

Of course... Make sure named & sshd work fine in Jail.

Scenario 4: Jail ssh server connect from FreeBSD Server internal
FreeBSD Server: Internal connect to Jail's sshd server(10.0.0.4)

Test result:
Code:
# /usr/local/bin/ssh 10.0.0.4
Permission denied (publickey).gth 668
It is OK, because authentication by public-private key pair.
SSHD work fine.

Scenario 5: Jail named server connect from FreeBSD Server internal
FreeBSD Server: Internal connect to Jail's named server(10.0.0.1)

Test result:
Code:
# nslookup
> server 10.0.0.1
Default server: 10.0.0.1
Address: 10.0.0.1#53
> epopen.com
Server:         10.0.0.1
Address:        10.0.0.1#53

Name:   epopen.com
Address: 123.194.237.215
Named work file.
My FreeBSD server's jailed server not work suddenly, via LAN also.
Possible PF's rdr not work I think, but I don't touch it
And "rdr" can setup state trace itself for jail connection return output, "nat" is not required from Google....(if mistake, why before work fine? confuse now)

Can help me debug?
Thanks a lot.
Neko.
 
I think these block traffic going in and out of your jails:
Code:
block in quick on $ext_if from { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> } to any
block out quick on $ext_if from any to { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> }

priv_net includes 10.0.0.0/8 with includes your jails.
 
SirDice said:
I think these block traffic going in and out of your jails:
Code:
block in quick on $ext_if from { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> } to any
block out quick on $ext_if from any to { <priv_net>, <realtimeBlockList>, <bruteforce>, <blockList-File>, <emerging-threats>, <jail_blockIP-Pool> }

priv_net includes 10.0.0.0/8 with includes your jails.
Thanks a lot.
But it is OK...
Rule1: Block source IP=private packet to server via WAN NIC
Rule2: Block destation IP=private packet to foreign via WAN NIC
These rules not work when normal operation.
Only hack packet activity.

In fact, I had been remove 10.0.0.0/8 and test, fail same.

Thanks a lot.
 
Update:
Before, Cable ISP issue possible I think...
So I moved my FreeBSD server go back ADSL (The ADSL used with FreeBSD server before 3 years, worked fine.)
And internal interface($int_if) not use.

Same issue, The root cause of issue from PF I think.
Can help me for debug?

Thanks a lot.
 
Update:
1.Added
Code:
nat on $ext_if inet from $jail_sshd_ip to any -> ($ext_if)
and test, won't work too.
..Here, I think # rdr create packet return path automatic when connection, so NO NEED # nat, Is correct?(Because worked fine without nat before...)
2.I tried PF rule
Code:
pass in log on $ext_if proto tcp from any to 10.0.0.4 port 22 keep state
, And # tcpdump -netvvi pflog0 on server and start# putty $SERVER_IP on windows got result at server
Code:
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
rule 38..16777216/0(match): pass in on fxp1: (tos 0x0, ttl 126, id 24416, offset 0, flags [DF], proto TCP (6), length 52)
    59.104.65.24.65091 > 10.0.0.4.22: Flags [S], cksum 0x759f (correct), seq 2550381711, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
..Only one packet (putty->server port ssh), but haven't any packet return to putty, And haven't anyone in sshd log, can't know sshd server received the first packet or not.

I tried run # tcpdump in jail but won't work too...

Possible any system configuration cause PF work fail?
Thanks a lot.
 
Try simplifying your ruleset for debugging purposes.

- Do not bundle too many items into one list or table. Try writing "too many" single purpose lines into your pf.conf.
- Deactivate/comment all features not vital, e.g. traffic shaping, etc.
- So start with minimum config to block all incoming, allow outgoing as needed
- Now activate all single rules (see first bullet point) one by one, don't try to solve all problems at once.
- Try to narrow down the problem (the only, single problem)
- Document your sections and your changes

As you are constantly changing and testing your config it's more like a moving target for others here to understand and help you.

Later on you can bundle expressions into tables, but first - keep it simple.
Also, consider using a minimum revision control (like rcs) to keep track of your changes in the config file and always have a way to go backup to previous versions.

(btw: IMHO I agree with SirDice about the blocking rules and your 10.x subnet)

Additional reading https://calomel.org/pf_config.html. My favorite ;)
 
Hi ALL

I followed gqgunhed's suggest, become minimum/simple config like under..
Code:
ext_if="fxp1"   # replace with actual external interface name i.e., dc0
int_if="fxp0"   # replace with actual internal interface name i.e., dc1

jail_named_ip="10.0.0.1"
jail_sshd_ip="10.0.0.4"

jail_named_port="domain"
jail_sshd_port="ssh"

nat on $ext_if inet from $jail_named_ip to any -> ($ext_if)
nat on $ext_if inet from $jail_sshd_ip to any -> ($ext_if)

rdr on $ext_if inet proto { udp, tcp } from any to ($ext_if) port { $jail_named_port } -> $jail_named_ip
rdr on $ext_if inet proto { tcp } from any to ($ext_if) port { $jail_sshd_port } -> $jail_sshd_ip

block log all

pass quick on { $ext_if } proto pfsync

pass out log quick on $ext_if proto udp from any to any port domain keep state

pass in log quick on $ext_if proto udp from any to $jail_named_ip port $jail_named_port keep state

pass out log quick on $ext_if proto tcp from any to any port { $jail_sshd_port } keep state
pass out log quick on $ext_if proto tcp from any to any port { $jail_named_port } keep state

# SSH rule
pass in log quick on $ext_if proto tcp from any to $jail_sshd_ip port { $jail_sshd_port } keep state
pass in log quick proto tcp from any to any port { 2222 } keep state

Name/ssh tested Same FAIL...

Note1: I had been use revision control (git) to manage all of my server configure. :D
About /etc/rc.conf, Least modified at 2012/09, about upstream speed change.

Note2: I tried add under in /etc/sysctl.conf
Code:
net.inet.ip.forwarding=1    # 1=Permit forwarding (routing) of packets
But won't work also.

May kernel/system configure issue?
Thanks a lot.
 
try using the redirects to SSH without {}-list entries, just with plain port numbers or the plain variable, just like so:
rdr on $ext_if inet proto { udp, tcp } from any to ($ext_if) port $jail_named_port -> $jail_named_ip
rdr on $ext_if inet proto { tcp } from any to ($ext_if) port $jail_sshd_port -> $jail_sshd_ip

AFAIK it doesn't make problems with the pass rules, but I think it is a problem within the redirects.
# SSH rule
pass in log quick on $ext_if proto tcp from any to $jail_sshd_ip port { $jail_sshd_port } keep state
pass in log quick proto tcp from any to any port { 2222 } keep state
 
Hi gqgunhed

I had been try your 2 suggest...

1.Remove {}-list under rule
Code:
rdr on $ext_if inet proto tcp from any to ($ext_if) port $jail_sshd_port -> $jail_sshd_ip
Test result fail.
Rule result(pfctl -sn) is:
Code:
rdr on fxp1 inet proto tcp from any to (fxp1) port = ssh -> 10.0.0.4
same {}-list existed.

2."Pass in" problem within the redirects
I tried to remove
Code:
pass in log quick on $ext_if proto tcp from any to $jail_sshd_ip port { $jail_sshd_port } keep state
Test result fail, and result from pflog0 under.
Code:
 00:00:04.158109 rule 0..16777216/0 ( match ) : block in on fxp1: 59.104.65.24.58747 > 10.0.0.4.22: Flags [S], seq 1760527666, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
"rdr" working with "pass in" I think..

Thanks a lot.
 
Hmm, strange.
I played around a little bit with my VMs here.
But it seems to work here.

I tried by setting up a fake listener with:
Code:
# ifconfig [nic] alias [ip-address] netmask 255.255.255.255
# while:; do nc -l [ip-adress] [port]; done;
where "ip-address" is the to-be-routed-to ip-address and "port" the to-be-routed-to port.

I watched the connections with sysutils/pftop and # tcpdump -n -i pflog.

I used # nc [ext-ip] [ext-port] to connect and was redirect and allowed-in to the listening netcat session.

(p.s. are you testing your external redirect from an external address - or from inside-out-back-to-inside?)
 
gqgunhed said:
Hmm, strange.
I played around a little bit with my VMs here.
But it seems to work here.

I tried by setting up a fake listener with:
Code:
# ifconfig [nic] alias [ip-address] netmask 255.255.255.255
# while:; do nc -l [ip-adress] [port]; done;
where "ip-address" is the to-be-routed-to ip-address and "port" the to-be-routed-to port.

I watched the connections with sysutils/pftop and # tcpdump -n -i pflog.

I used # nc [ext-ip] [ext-port] to connect and was redirect and allowed-in to the listening netcat session.

(p.s. are you testing your external redirect from an external address - or from inside-out-back-to-inside?)

Thanks you a lot.
I strange too, because server worked before 1 month...
System configure un-change but work fail suddenly..

About...
1.external redirect from an external address
Yes, WAN and test machine is different ISP.
2.inside-out-back-to-inside
Is mean from host to jail?

From jail to internet server (added "nat" rule) won't work also.
rdr and nat won't work both.
It look like whole system problem relation pf redirect packet...

I plan buy new hardware and install new system at next month.
But can't discovery root cause in the issue...
It cause problem in new system possible and can't fix it.

Thanks a lot.
 
Hi all...

I give the issue, System corrupt I think.
I can't believe it, but it is true.

Thanks a lot.
 
Hi all,

Absolute root cause is system corruption I think. Please reference below scenario:

Code:
Scenario1: Previous work fine.
   External NIC -> PF -> daemon in jail : OK
   Host -> daemon in jail : OK

Scenario2: The issue.
   External NIC -> PF -> daemon in jail : NG
   Host -> daemon in jail : OK

Scenario3: 3 days ago and today 2 times.
   External NIC -> PF -> daemon in jail : NG
   Host -> daemon in jail : NG

In scenario3, the host won't local connect to to 2 jail (mysql-server and sshd), restarting jail won't fix. Both daemons work fine (no error log) in jail look like. Only after reboot the host system is re-connectable.

I don't know why even now, FreeBSD's bug?
Thanks a lot.
 
I'm not sure if polling is a recommended workaround on any recent machine. I think there was a thread on the mailing lists saying that polling is a strongly discouraged method now with MSI and MSI-X interrupts. Can't find it now.
 
Toast said:
Add "-rxcsum" to fxp1. :)
Code:
ifconfig_fxp1="DHCP polling -rxcsum"
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170081

Solved! Your solution works! Thanks you very much.
Now working ifconfig_fxp1 in rc.conf:
Code:
ifconfig_fxp1="DHCP polling -rxcsum -rxcsum6"
But I'm confused and have questions:
  1. The issue occurred after a system reboot, but I didn't touch anything before the reboot, kernel/world/driver weren't changed, so why did it occur suddenly? (Later, the upgrade to 9.1-STABLE didn't work either).
  2. Disabling checksum offloading works, does that mean that the fxp1 checksum offloading hardware engine has a breakdown/defect?
  3. If offloading the hardware engine is really broken down, why does connecting to the jail fail, but to the host work OK?
  4. Does the kernel/driver load increase when checksum offloading is disabled?
  5. PR kern/170081 has no maintainer from July 2012, Tiny issue/unfixable/hardware defect possible?

Anyway, thanks everyone very much who posted in the thread.
 
kpa said:
I'm not sure if polling is a recommended workaround on any recent machine. I think there was a thread on the mailing lists saying that polling is a strongly discouraged method now with MSI and MSI-X interrupts. Can't find it now.

Thanks for your suggestion. I tried to remove the "polling" option from ifconfig_fxp1, but it does not work the same.

Thanks you very much.
 
Back
Top