Hi everyone, recently I've faced a strange issue with a jail on my FreeBSD router, and hope that someone could give me a hand in solving it.
Month or two ago I had hardware issue with my router when HDDs were disconnected because of motherboard chipset overheat. After solving it I've found that my only jail lost internet connection, ping and traceroute worked just fine, everything else is not. I've thought that some jail files got corrupted during failures above, and created a new test jail according to this manual, but it has just the same issue with connectivity. I've decided that maybe some host system files got corrupted, and today finally get some time to reinstall host OS only to find that issue still there. I've minimized both configuration and firewall rules to reproduce the issue, here are details for relevant parts:
uname -a
ifconfig
jail.conf
rc.conf
Mini firewall
Pinging test host from jail (i.ua selected as convenient short website name for testing)
Looks good, checking firewall log
Trying to fetch site main page
No luck, but firewall log looks completely the same
So response successfully passed NAT and was accepted by firewall, but somehow didn't reach the jail.
Fetch from host system works just fine
I've used tcpdump to compare both fetches
Host:
Jail:
Same picture as with firewall logs above, tcpdump sees response from web site, but it doesn't reach the jail.
Does anybody have any idea what could be wrong here, or how this could be debugged further?
Month or two ago I had hardware issue with my router when HDDs were disconnected because of motherboard chipset overheat. After solving it I've found that my only jail lost internet connection, ping and traceroute worked just fine, everything else is not. I've thought that some jail files got corrupted during failures above, and created a new test jail according to this manual, but it has just the same issue with connectivity. I've decided that maybe some host system files got corrupted, and today finally get some time to reinstall host OS only to find that issue still there. I've minimized both configuration and firewall rules to reproduce the issue, here are details for relevant parts:
uname -a
Bash:
FreeBSD bsd-route 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:09:26 UTC 2018 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386
Bash:
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2009<RXCSUM,VLAN_MTU,WOL_MAGIC>
ether 44:8a:5b:9f:c5:8e
hwaddr 00:90:27:77:aa:6a
inet 100.66.1.248 netmask 0xfffff000 broadcast 100.66.15.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:07:e9:10:9d:79
hwaddr 00:07:e9:10:9d:79
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8210b<RXCSUM,TXCSUM,VLAN_MTU,TSO4,WOL_MAGIC,LINKSTATE>
ether 00:15:f2:7b:ae:5a
hwaddr 00:15:f2:7b:ae:5a
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (none)
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:22:b0:01:aa:be
hwaddr 00:22:b0:01:aa:be
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
status: running
ssid myhomeap channel 2 (2417 MHz 11g) bssid 00:22:b0:01:aa:be
regdomain FCC country US authmode WPA2/802.11i privacy MIXED
deftxkey 2 AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 30 scanvalid 60
protmode CTS dtimperiod 1 -dfs
groups: wlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:89:95:37:fd:00
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
inet6 fe80::89:95ff:fe37:fd00%bridge0 prefixlen 64 scopeid 0x6
inet6 fc00:172:16:1::1 prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 370370
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 2000000
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 172.16.2.1 netmask 0xffffff00
inet 172.16.2.2 netmask 0xffffffff
inet 172.16.2.3 netmask 0xffffffff
inet6 fe80::1%lo1 prefixlen 64 scopeid 0x7
inet6 fc00:172:16:1:1000::1 prefixlen 64
inet6 fc00:172:16:1:1000::2 prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
Bash:
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = "/usr/px/Jails/$name";
test {
host.hostname = "test";
ip4.addr = 172.16.2.3;
allow.raw_sockets;
}
Bash:
ifconfig_lo1="inet 172.16.2.1 netmask 255.255.255.0" # Jails GW
ifconfig_lo1_alias0="inet 172.16.2.2 netmask 255.255.255.255" # First jail alias
ifconfig_lo1_alias1="inet 172.16.2.3 netmask 255.255.255.255" # Test jail alias
...
jail_enable="YES"
gateway_enable="YES"
Bash:
#!/bin/sh
FwCMD="/sbin/ipfw"
LanOut="fxp0"
IpOut="100.66.1.248"
NetMask="24"
NetIn="172.16.1.0"
JailIn="172.16.2.0"
# Flush everything
${FwCMD} -f flush
${FwCMD} -f pipe flush
${FwCMD} -f queue flush
${FwCMD} -f zero
# NAT
${FwCMD} nat 100 config if ${LanOut} log same_ports unreg_only
${FwCMD} add nat 100 log all from ${NetIn}/${NetMask} to any out via ${LanOut}
${FwCMD} add nat 100 log all from ${JailIn}/${NetMask} to any out via ${LanOut}
${FwCMD} add nat 100 log all from any to ${IpOut} in via ${LanOut}
# Allow everything
${FwCMD} add allow log logamount 100000 ip from any to any
Bash:
# ping i.ua
PING i.ua (91.198.36.14): 56 data bytes
64 bytes from 91.198.36.14: icmp_seq=0 ttl=60 time=1.393 ms
64 bytes from 91.198.36.14: icmp_seq=1 ttl=60 time=1.375 ms
Bash:
Oct 9 22:45:51 bsd-route kernel: ipfw: 200 Nat ICMP:8.0 172.16.2.3 91.198.36.14 out via fxp0
Oct 9 22:45:51 bsd-route kernel: ipfw: 400 Accept ICMP:8.0 100.66.1.248 91.198.36.14 out via fxp0
Oct 9 22:45:51 bsd-route kernel: ipfw: 300 Nat ICMP:0.0 91.198.36.14 100.66.1.248 in via fxp0
Oct 9 22:45:51 bsd-route kernel: ipfw: 400 Accept ICMP:0.0 91.198.36.14 172.16.2.3 in via fxp0
Bash:
# fetch http://i.ua
fetch: http://i.ua: Operation timed out
Bash:
Oct 9 22:46:02 bsd-route kernel: ipfw: 200 Nat TCP 172.16.2.3:28522 91.198.36.14:80 out via fxp0
Oct 9 22:46:02 bsd-route kernel: ipfw: 400 Accept TCP 100.66.1.248:28522 91.198.36.14:80 out via fxp0
Oct 9 22:46:02 bsd-route kernel: ipfw: 300 Nat TCP 91.198.36.14:80 100.66.1.248:28522 in via fxp0
Oct 9 22:46:02 bsd-route kernel: ipfw: 400 Accept TCP 91.198.36.14:80 172.16.2.3:28522 in via fxp0Fetch from host system works just fine
Bash:
# fetch http://i.ua
Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
673056224:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/s3_clnt.c:1269:
fetch: http://i.ua: Authentication errorHost:
Bash:
17:12:46.862095 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
100.66.1.248.13291 > www.i.ua.http: Flags [S], cksum 0x89b7 (correct), seq 2503970007, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 8290667 ecr 0], length 0
17:12:46.863418 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
www.i.ua.http > 100.66.1.248.13291: Flags [S.], cksum 0xcc90 (correct), seq 2528115947, ack 2503970008, win 5792, options [mss 1460,sackOK,TS val 3705681400 ecr 8290667,nop,wscale 7], length 0
17:12:46.863481 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
100.66.1.248.13291 > www.i.ua.http: Flags [.], cksum 0x0df9 (correct), seq 1, ack 1, win 1026, options [nop,nop,TS val 8290669 ecr 3705681400], length 0
17:12:46.864168 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 146)
100.66.1.248.13291 > www.i.ua.http: Flags [P.], cksum 0x5606 (correct), seq 1:95, ack 1, win 1026, options [nop,nop,TS val 8290669 ecr 3705681400], length 94: HTTP, length: 94
GET / HTTP/1.1
Host: i.ua
Accept: */*
User-Agent: fetch libfetch/2.0
Connection: close
Bash:
16:47:54.768143 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x9e0e (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6798573 ecr 0], length 0
16:47:54.769465 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0x03b1 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704189202 ecr 6798573,nop,wscale 7], length 0
16:47:57.767610 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x9256 (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6801573 ecr 0], length 0
16:47:57.769038 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0xf7f9 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704192201 ecr 6798573,nop,wscale 7], length 0
16:47:58.973410 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
www.i.ua.http > 100.66.1.248.41436: Flags [S.], cksum 0xf344 (correct), seq 941882795, ack 794375435, win 5792, options [mss 1460,sackOK,TS val 3704193406 ecr 6798573,nop,wscale 7], length 0
16:48:00.967617 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
100.66.1.248.41436 > www.i.ua.http: Flags [S], cksum 0x85d6 (correct), seq 794375434, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 6804773 ecr 0], length 0Does anybody have any idea what could be wrong here, or how this could be debugged further?