Solved ipfw vs. ping puzzle

FKEinternet

FKEinternet

My servers Dreamer and Wren each have two interfaces, connected to two routers. The re0 interfaces are connected to the 192.168.14.* subnet, and the re1 interfaces are connected to the 192.168.1.* subnet. The 192.168.1.* subnet originates at a Verizon router, which is also upstream from an Asus router where the 192.168.14.* subnet originates.

I can ping() Dreamer from Wren on 192.168.14.199 any time, whether ipfw() is running or not. However, if I try to ping() Dreamer from Wren on 192.168.1.199, it only works when ipfw() is not running. That points a really suspicious finger at my ipfw() configuration, and I've gone through so many rule combinations trying to get this to work I'm at a loss to figure it out.

Ping success looks like this:
Code: 
# ping 192.168.14.199
PING 192.168.14.199 (192.168.14.199): 56 data bytes
64 bytes from 192.168.14.199: icmp_seq=0 ttl=64 time=0.204 ms
64 bytes from 192.168.14.199: icmp_seq=1 ttl=64 time=0.188 ms
64 bytes from 192.168.14.199: icmp_seq=2 ttl=64 time=0.162 ms
64 bytes from 192.168.14.199: icmp_seq=3 ttl=64 time=0.160 ms
64 bytes from 192.168.14.199: icmp_seq=4 ttl=64 time=0.176 ms
64 bytes from 192.168.14.199: icmp_seq=5 ttl=64 time=0.176 ms
^C
--- 192.168.14.199 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.160/0.178/0.204/0.015 ms
Ping failure looks like this:
Code: 
# ping 192.168.1.199
PING 192.168.1.199 (192.168.1.199): 56 data bytes
^C
--- 192.168.1.199 ping statistics ---
239 packets transmitted, 0 packets received, 100.0% packet loss
This is the ping() target (Dreamer) that is not responding as expected:
Code: 
# uname -a
FreeBSD Dreamer.FKEinternet.net 10.2-RELEASE FreeBSD 10.2-RELEASE #0: Mon Oct  5 23:53:36 EDT 2015  root@Dreamer.FKEinternet.com.:/usr/obj/usr/src/sys/GENERIC  amd64

Right now I have these rules in the Dreamer (ping() target) ipfw() configuration:
Code: 
# ipfw list | grep check
00101 check-state log

# ipfw list | grep icmp
00235 allow log icmp from me to any icmptypes 8 via re0
00236 allow log icmp from me to any icmptypes 8 via re1
00237 allow log icmp from any to me icmptypes 0 via re0
00238 allow log icmp from any to me icmptypes 0 via re1
00239 allow log icmp from any to me icmptypes 11 via re0
00240 allow log icmp from any to me icmptypes 11 via re1
00250 allow log icmp from any to any out via re0 keep-state
00251 allow log icmp from any to any out via re1 keep-state
00510 allow log icmp from any to any in via re0
00511 allow log icmp from any to any in via re1
In Dreamer's /var/log/security, I see entries such as these when ping() is failing on 192.168.1.199:
Code: 
Mar 26 10:53:25 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
Mar 26 10:53:26 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
Mar 26 10:53:27 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
Mar 26 10:53:28 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
Mar 26 10:53:29 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
Mar 26 10:53:30 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.1.201 192.168.1.199 in via re1
When ping() works on 192.168.14.199, I see entries such as these in Dreamer's /var/log/security:
Code: 
Mar 26 11:05:00 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:8.0 192.168.14.201 192.168.14.199 in via re0
Mar 26 11:05:00 Dreamer kernel: ipfw: 250 Accept ICMP:8.0 192.168.14.201 192.168.14.199 in via re0
Mar 26 11:05:00 Dreamer kernel: ipfw: 101 UNKNOWN ICMP:0.0 192.168.14.199 192.168.14.201 out via re0
Mar 26 11:05:00 Dreamer kernel: ipfw: 250 Accept ICMP:0.0 192.168.14.199 192.168.14.201 out via re0
In /etc/inetd.conf on Dreamer (the ping() target) I have
Code: 
echo  stream  tcp nowait  root  internal  -l
echo  stream  tcp6  nowait  root  internal  -l
echo  dgram  udp wait  root  internal  -l
echo  dgram  udp6  wait  root  internal  -l
(I added the "-l" flags to try to get logging to occur when adding the flag in /etc/rc.conf had no effect - and it didn't do anything here, either.)

In /etc/rc.conf on Dreamer I have
Code: 
hostname="Dreamer.FKEinternet.net"
defaultrouter="192.168.1.1"
#
ifconfig_re0="inet 192.168.14.69 netmask 255.255.252.0"
ifconfig_re0_alias0="inet 192.168.14.253 netmask 255.255.252.0"
ifconfig_re0_alias1="inet 192.168.14.199 netmask 255.255.252.0"
ifconfig_re0_alias2="inet 192.168.14.202 netmask 255.255.252.0"
#
ifconfig_re1="inet 100.0.193.99 netmask 255.255.255.0"
ifconfig_re1_alias0="inet 192.168.1.199 netmask 255.255.255.0"
ifconfig_re1_alias1="inet 100.0.193.102 netmask 255.255.255.0"
ifconfig_re1_alias2="inet 192.168.1.202 netmask 255.255.255.0"
#
ifconfig_lo0_alias0="inet 127.0.0.2 netmask 255.255.255.255"  # ns2 for second domain
#
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"
firewall_logging="YES"

inetd_enable="YES"
inetd_flags="-l -d"
where adding the "-d" flag apparently prevented inetd() from disconnecting from the console, and generated this output:
Code: 
# service inetd restart
Stopping inetd.
Waiting for PIDS: 12630.
Starting inetd.
ADD : pop3 proto=tcp accept=1 max=0 user=root group=(null)class=daemon builtin=0x0 server=/usr/local/libexec/qpopper policy=""
inetd: pop3/tcp: ipsec initialization failed; in entrust
inetd: pop3/tcp: ipsec initialization failed; out entrust
inetd: enabling pop3, fd 4
inetd: registered /usr/local/libexec/qpopper on 4
ADD : pop3s proto=tcp accept=1 max=0 user=root group=(null)class=daemon builtin=0x0 server=/usr/local/libexec/qpopper policy=""
inetd: pop3s/tcp: ipsec initialization failed; in entrust
inetd: pop3s/tcp: ipsec initialization failed; out entrust
inetd: enabling pop3s, fd 5
inetd: registered /usr/local/libexec/qpopper on 5
ADD : echo proto=tcp accept=1 max=0 user=root group=(null)class=daemon builtin=0x60b6f0 server=internal policy=""
inetd: echo/tcp: ipsec initialization failed; in entrust
inetd: echo/tcp: ipsec initialization failed; out entrust
inetd: enabling echo, fd 6
inetd: registered internal on 6
ADD : echo proto=tcp accept=1 max=0 user=root group=(null)class=daemon builtin=0x60b6f0 server=internal policy=""
inetd: echo/tcp: ipsec initialization failed; in entrust
inetd: echo/tcp: ipsec initialization failed; out entrust
inetd: enabling echo, fd 7
inetd: registered internal on 7
ADD : echo proto=udp accept=1 max=1 user=root group=(null)class=daemon builtin=0x60b710 server=internal policy=""
inetd: echo/udp: ipsec initialization failed; in entrust
inetd: echo/udp: ipsec initialization failed; out entrust
inetd: enabling echo, fd 8
inetd: registered internal on 8
ADD : echo proto=udp accept=1 max=1 user=root group=(null)class=daemon builtin=0x60b710 server=internal policy=""
inetd: echo/udp: ipsec initialization failed; in entrust
inetd: echo/udp: ipsec initialization failed; out entrust
inetd: enabling echo, fd 9
inetd: registered internal on 9
(The inetd() "-d" flag was a temporary experiment, not present during most of my testing, and now permanently removed.)

Are my ipfw() rules for ICMP wrong, or is there something else going on that I'm not properly taking into account?

For bonus points, where is the inetd() logging output going? ... and is there any documentation for the format and contents of the ipfw() log entries in /var/log/security (e.g., what do all of those UNKNOWN marks mean)?
 
Thinking there might be an internal affinity for the first network interface, I swapped the network cables on Dreamer and rewrote /etc/rc.conf to match:
Code: 
hostname="Dreamer.FKEinternet.net"
defaultrouter="192.168.1.1"
#
ifconfig_re0="inet 100.0.193.99 netmask 255.255.255.0"
ifconfig_re0_alias0="inet 192.168.1.199 netmask 255.255.255.0"
ifconfig_re0_alias1="inet 100.0.193.102 netmask 255.255.255.0"
ifconfig_re0_alias2="inet 192.168.1.202 netmask 255.255.255.0"
#
ifconfig_re1="inet 192.168.14.69 netmask 255.255.252.0"
ifconfig_re1_alias0="inet 192.168.14.253 netmask 255.255.252.0"
ifconfig_re1_alias1="inet 192.168.14.199 netmask 255.255.252.0
ifconfig_re1_alias2="inet 192.168.14.202 netmask 255.255.252.0"
I got the same results: ping() from Wren worked on 192.168.14.199, and not on 192.168.1.199.

After pondering the UNKNOWN tag usually present in the check-state log entries in /var/log/security, I realized it's because all packets arriving at the firewall are being checked by that rule and logged, and new ones that haven't been through the system are "unknown" since they don't match anything in the dynamic rules. So, where were all of the packets going? They got into the firewall, but then never heard from again in most cases. All of a sudden it occurred to me what was going on - the packets were being handled before they got to the default rules that would log their being discarded - handled by rules that don't have logging turned on. Obviously the ping() packets were being handled by a rule I wasn't expecting them to get to.

It turns out the ping() problem was this set of rules:
Code: 
# inbound rules
$cmd 00495 allow all from 127.0.0.0/8 to me in via $if0     #loopback
$cmd 00495 allow all from 127.0.0.0/8 to me in via $if1     #loopback
$cmd 00498 allow all from 192.168.14.0/24 to me in via $if0   #local network
$cmd 00498 allow all from 192.168.14.0/24 to me in via $if1   #local network
$cmd 00499 allow all from 192.168.12.0/24 to me in via $if0   #local network
$cmd 00499 allow all from 192.168.12.0/24 to me in via $if1   #local network
# ========================================================================================
# deny inbound traffic to restricted addresses
$cmd 00500 deny all from 192.168.0.0/16 to any in via $if0   #RFC 1918 private IP
$cmd 00500 deny all from 192.168.0.0/16 to any in via $if1   #RFC 1918 private IP
I'm using more than just 127.0.0.1 for loopback, so I needed to expand the loopback range that's allowed. I added the rule(s) to allow the 192.168.14.* traffic because the Verizon router is doing static NAT forwarding of my public IP addresses, so the local network range has to be allowed. I later added the 192.168.12.* rule(s) to support the laptop I use for my day job. Any other 192.168.*.* traffic is bogus, because those IPs can't come across the Internet - except that I'd rearranged my network topology, and now there's another subnet that has to be allowed 192.168.1.*! Once I added a pair of rules to allow incoming packets on that subnet, ping() started working the way I expected it to.

I wasn't out of the woods yet, though: I still couldn't get a Web page from the server, and I couldn't receive mail. I looked at /var/log/maillog and discovered that sendmail was in fact receiving mail - the problem was actually that Thunderbird on my desktop couldn't connect to collect the mail. I set up a filter, and made an enlightening discovery:
Code: 
[root]pts/0://Dreamer/etc/mail # tail -f /var/log/security | grep 995
Mar 27 21:39:39 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.1:3751 192.168.1.199:995 in via re0
Mar 27 21:39:39 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
Mar 27 21:39:39 Dreamer kernel: ipfw: 299 Deny TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
Mar 27 21:39:42 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.1:3751 192.168.1.199:995 in via re0
Mar 27 21:39:42 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
Mar 27 21:39:42 Dreamer kernel: ipfw: 299 Deny TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
Mar 27 21:39:49 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.1:3751 192.168.1.199:995 in via re0
Mar 27 21:39:49 Dreamer kernel: ipfw: 101 UNKNOWN TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
Mar 27 21:39:49 Dreamer kernel: ipfw: 299 Deny TCP 192.168.1.199:995 192.168.1.1:3751 out via re0
The packets where Thunderbird was trying to connect were being received, but the responses were being blocked. Looking once again at the same region in the filter rules, I realized what the problem was: I had to add the keep-state option on the inbound local network rules so the replies could be returned.
Code: 
# otherwise deny outbound packets
# outbound catchall.
$cmd 00299 deny log ip from any to any out via $if0
$cmd 00299 deny log ip from any to any out via $if1

# inbound rules
$cmd 00495 allow all from 127.0.0.0/8 to me in via $if0 keep-state   #loopback
$cmd 00495 allow all from 127.0.0.0/8 to me in via $if1 keep-state   #loopback
$cmd 00498 allow all from 192.168.1.0/24 to me in via $if0 keep-state   #local network
$cmd 00498 allow all from 192.168.1.0/24 to me in via $if1 keep-state   #local network
$cmd 00498 allow all from 192.168.14.0/24 to me in via $if0 keep-state   #local network
$cmd 00498 allow all from 192.168.14.0/24 to me in via $if1 keep-state   #local network
$cmd 00499 allow all from 192.168.12.0/24 to me in via $if0 keep-state   #local network
$cmd 00499 allow all from 192.168.12.0/24 to me in via $if1 keep-state   #local network
# ========================================================================================
# deny inbound traffic to restricted addresses
$cmd 00500 deny log all from 192.168.0.0/16 to any in via $if0   #RFC 1918 private IP
$cmd 00500 deny log all from 192.168.0.0/16 to any in via $if1   #RFC 1918 private IP
I also added the log flag to the 192.168.*.* blocking rules so that if I change my internal network topology again in the future I won't have to spend as much time trying to figure out I need to add a new subnet to the allowed range.

Now I can ping() Dreamer on all of the internal network interfaces, I can load Web pages from my server, and Thunderbird is collecting mail for my domains once again: I'm finally back to where I was a week and a half ago, except that now I've got options for getting rid of the Verizon router that's allowing NetBIOS packets to leak onto the public Internet (and probably letting bogus NetBIOS packets in).

I never thought I would see the day when SPAM was welcome in my inbox - but it's telling me that the email system is working, so for the moment I'm not terribly upset by it...
 
You must log in or register to reply here.
Back
Top