Solved Internet traffic doesn't return to jail, but icmp works fine

-Px-

New Member


Messages: 2

#1
Hi everyone, recently I've faced a strange issue with a jail on my FreeBSD router, and hope that someone could give me a hand in solving it.
Month or two ago I had hardware issue with my router when HDDs were disconnected because of motherboard chipset overheat. After solving it I've found that my only jail lost internet connection, ping and traceroute worked just fine, everything else is not. I've thought that some jail files got corrupted during failures above, and created a new test jail according to this manual, but it has just the same issue with connectivity. I've decided that maybe some host system files got corrupted, and today finally get some time to reinstall host OS only to find that issue still there. I've minimized both configuration and firewall rules to reproduce the issue, here are details for relevant parts:
uname -a
Bash:
FreeBSD bsd-route 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:09:26 UTC 2018     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386
ifconfig
Bash:
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2009<RXCSUM,VLAN_MTU,WOL_MAGIC>
        ether 44:8a:5b:9f:c5:8e
        hwaddr 00:90:27:77:aa:6a
        inet 100.66.1.248 netmask 0xfffff000 broadcast 100.66.15.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:07:e9:10:9d:79
        hwaddr 00:07:e9:10:9d:79
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8210b<RXCSUM,TXCSUM,VLAN_MTU,TSO4,WOL_MAGIC,LINKSTATE>
        ether 00:15:f2:7b:ae:5a
        hwaddr 00:15:f2:7b:ae:5a
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (none)
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:22:b0:01:aa:be
        hwaddr 00:22:b0:01:aa:be
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
        status: running
        ssid myhomeap channel 2 (2417 MHz 11g) bssid 00:22:b0:01:aa:be
        regdomain FCC country US authmode WPA2/802.11i privacy MIXED
        deftxkey 2 AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 30 scanvalid 60
        protmode CTS dtimperiod 1 -dfs
        groups: wlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:89:95:37:fd:00
        inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
        inet6 fe80::89:95ff:fe37:fd00%bridge0 prefixlen 64 scopeid 0x6
        inet6 fc00:172:16:1::1 prefixlen 64
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 370370
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 172.16.2.1 netmask 0xffffff00
        inet 172.16.2.2 netmask 0xffffffff
        inet 172.16.2.3 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x7
        inet6 fc00:172:16:1:1000::1 prefixlen 64
        inet6 fc00:172:16:1:1000::2 prefixlen 64
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
jail.conf
Bash:
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = "/usr/px/Jails/$name";

test {
        host.hostname = "test";
        ip4.addr = 172.16.2.3;
        allow.raw_sockets;
}
rc.conf
Bash:
ifconfig_lo1="inet 172.16.2.1 netmask 255.255.255.0" # Jails GW
ifconfig_lo1_alias0="inet 172.16.2.2 netmask 255.255.255.255" # First jail alias
ifconfig_lo1_alias1="inet 172.16.2.3 netmask 255.255.255.255" # Test jail alias
...
jail_enable="YES"
gateway_enable="YES"
Mini firewall
Bash:
#!/bin/sh

FwCMD="/sbin/ipfw"
LanOut="fxp0"
IpOut="100.66.1.248"
NetMask="24"
NetIn="172.16.1.0"
JailIn="172.16.2.0"

# Flush everything
${FwCMD} -f flush
${FwCMD} -f pipe flush
${FwCMD} -f queue flush
${FwCMD} -f zero

# NAT
${FwCMD} nat 100 config if ${LanOut} log same_ports unreg_only
${FwCMD} add nat 100 log all from ${NetIn}/${NetMask} to any out via ${LanOut}
${FwCMD} add nat 100 log all from ${JailIn}/${NetMask} to any out via ${LanOut}
${FwCMD} add nat 100 log all from any to ${IpOut} in via ${LanOut}

# Allow everything
${FwCMD} add allow log logamount 100000 ip from any to any
Pinging test host from jail (i.ua selected as convenient short website name for testing)
Bash:
# ping i.ua
PING i.ua (91.198.36.14): 56 data bytes
64 bytes from 91.198.36.14: icmp_seq=0 ttl=60 time=1.393 ms
64 bytes from 91.198.36.14: icmp_seq=1 ttl=60 time=1.375 ms
Looks good, checking firewall log
Bash:
Oct  9 22:45:51 bsd-route kernel: ipfw: 200 Nat ICMP:8.0 172.16.2.3 91.198.36.14 out via fxp0
Oct  9 22:45:51 bsd-route kernel: ipfw: 400 Accept ICMP:8.0 100.66.1.248 91.198.36.14 out via fxp0
Oct  9 22:45:51 bsd-route kernel: ipfw: 300 Nat ICMP:0.0 91.198.36.14 100.66.1.248 in via fxp0
Oct  9 22:45:51 bsd-route kernel: ipfw: 400 Accept ICMP:0.0 91.198.36.14 172.16.2.3 in via fxp0
Trying to fetch site main page
Bash:
# fetch http://i.ua
fetch: http://i.ua: Operation timed out
No luck, but firewall log looks completely the same
Bash:
Oct  9 22:46:02 bsd-route kernel: ipfw: 200 Nat TCP 172.16.2.3:28522 91.198.36.14:80 out via fxp0
Oct  9 22:46:02 bsd-route kernel: ipfw: 400 Accept TCP 100.66.1.248:28522 91.198.36.14:80 out via fxp0
Oct  9 22:46:02 bsd-route kernel: ipfw: 300 Nat TCP 91.198.36.14:80 100.66.1.248:28522 in via fxp0
Oct  9 22:46:02 bsd-route kernel: ipfw: 400 Accept TCP 91.198.36.14:80 172.16.2.3:28522 in via fxp0
So response successfully passed NAT and was accepted by firewall, but somehow didn't reach the jail.
Fetch from host system works just fine
Bash:
# fetch http://i.ua
Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
673056224:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/s3_clnt.c:1269:
fetch: http://i.ua: Authentication error
I've used tcpdump to compare both fetches
Host:
Bash:
17:12:46.862095 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    100.66.1.248.13291 > www.i.ua.http: Flags [S], cksum 0x89b7 (correct), seq 2503970007, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 8290667 ecr 0], length 0
17:12:46.863418 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    www.i.ua.http > 100.66.1.248.13291: Flags [S.], cksum 0xcc90 (correct), seq 2528115947, ack 2503970008, win 5792, options [mss 1460,sackOK,TS val 3705681400 ecr 8290667,nop,wscale 7], length 0
17:12:46.863481 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    100.66.1.248.13291 > www.i.ua.http: Flags [.], cksum 0x0df9 (correct), seq 1, ack 1, win 1026, options [nop,nop,TS val 8290669 ecr 3705681400], length 0
17:12:46.864168 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 146)
    100.66.1.248.13291 > www.i.ua.http: Flags [P.], cksum 0x5606 (correct), seq 1:95, ack 1, win 1026, options [nop,nop,TS val 8290669 ecr 3705681400], length 94: HTTP, length: 94
        GET / HTTP/1.1
        Host: i.ua
        Accept: */*
        User-Agent: fetch libfetch/2.0
        Connection: close
Jail:
Bash:
16:47:54.768143 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x9e0e (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6798573 ecr 0], length 0
16:47:54.769465 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0x03b1 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704189202 ecr 6798573,nop,wscale 7], length 0
16:47:57.767610 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x9256 (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6801573 ecr 0], length 0
16:47:57.769038 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0xf7f9 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704192201 ecr 6798573,nop,wscale 7], length 0
16:47:58.973410 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0xf344 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704193406 ecr 6798573,nop,wscale 7], length 0
16:48:00.967617 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x85d6 (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6804773 ecr 0], length 0
Same picture as with firewall logs above, tcpdump sees response from web site, but it doesn't reach the jail.

Does anybody have any idea what could be wrong here, or how this could be debugged further? :)
 
Top