Solved [IPFW] DNS blocked for FreeBSD.org

As of yesterday afternoon I stopped receiving emails from the freebsd.org mailing lists on my personal domain/network and upon further digging today found that I was unable to access freebsd.org or any of its sub-domains. Near as I can tell I had/have no problems with any other domains and after "digging" into this for freebsd.org I found the problem to be DNS related. I simply changed my primary DNS to an external system and all works. Problem averted, but I still want to know why it stopped working and how I can fix it internally.

So, let's start with what I've determined. First off, this is happening to all four of my FreeBSD 9.3-RELEASE-p4 systems on my personal network that are dual stacked with public IPv4 and IPv6 addresses. When I do a dig +trace freebsd.org from my network, I get through all the nameservers until the final A record, which times out (AAAA also times out):
Code:
{17}root@hydra(~)> dig +trace freebsd.org

; <<>> DiG 9.9.5 <<>> +trace freebsd.org
;; global options: +cmd
.       3600000   IN   NS   M.ROOT-SERVERS.NET.
.       3600000   IN   NS   K.ROOT-SERVERS.NET.
.       3600000   IN   NS   B.ROOT-SERVERS.NET.
.       3600000   IN   NS   E.ROOT-SERVERS.NET.
.       3600000   IN   NS   J.ROOT-SERVERS.NET.
.       3600000   IN   NS   H.ROOT-SERVERS.NET.
.       3600000   IN   NS   D.ROOT-SERVERS.NET.
.       3600000   IN   NS   F.ROOT-SERVERS.NET.
.       3600000   IN   NS   A.ROOT-SERVERS.NET.
.       3600000   IN   NS   C.ROOT-SERVERS.NET.
.       3600000   IN   NS   I.ROOT-SERVERS.NET.
.       3600000   IN   NS   G.ROOT-SERVERS.NET.
.       3600000   IN   NS   L.ROOT-SERVERS.NET.
;; Received 797 bytes from 206.162.192.2#53(206.162.192.2) in 54 ms

org.       172800   IN   NS   a2.org.afilias-nst.info.
org.       172800   IN   NS   d0.org.afilias-nst.org.
org.       172800   IN   NS   a0.org.afilias-nst.info.
org.       172800   IN   NS   b2.org.afilias-nst.org.
org.       172800   IN   NS   b0.org.afilias-nst.org.
org.       172800   IN   NS   c0.org.afilias-nst.info.
org.       86400   IN   DS   21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org.       86400   IN   DS   21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0 D90F01BA
org.       86400   IN   RRSIG   DS 8 1 86400 20141106050000 20141030040000 22603 . BNPULda5SeKowJUHPxSXRPv9Cwc/l/u3iGD8cfg1YrF71uIxOzBkPYpf qeHXIPCkBMAGGMNPyjMAv/sdF7eykDVSGC1swerDdF+2qRUrA7lM6wc8 Tud/4Vk3Q80V4lAxnLW2ApfnXOaljkrbVBlAzwVC3D30tRgL4S9obQAU BRY=
;; Received 685 bytes from 192.228.79.201#53(B.ROOT-SERVERS.NET) in 95 ms

freebsd.org.     86400   IN   NS   ns2.isc-sns.com.
freebsd.org.     86400   IN   NS   ns1.isc-sns.net.
freebsd.org.     86400   IN   NS   ns3.isc-sns.info.
freebsd.org.     86400   IN   DS   32659 8 2 AF3B32E46DF2FC32C0110C7D6B808EE73E0411501AFAF9022D3DCD0A FA5B3ACD
freebsd.org.     86400   IN   RRSIG   DS 7 2 86400 20141115155009 20141025145009 57479 org. SDAo+3o5ILkTI1fKUmQW/EsOU3nNKiDlWcxweiX5ND8OLABiy7Bx6YYd KknL6dEQB7NJq50RELLQY5JfbXsf5ormH9bgswOHjS+bVbosSRTiHdnx Mcp20/H6Tkw8V0VXkHhSGvbZ+Zh2m30NGv7PjM1nTbiSINWutgvRY2CR 0wQ=
;; Received 339 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 45 ms

;; connection timed out; no servers could be reached

Yet a direct query against those last nameservers does work?
Code:
{33}root@hydra(~)> dig freebsd.org @ns1.isc-sns.net

; <<>> DiG 9.9.5 <<>> freebsd.org @ns1.isc-sns.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56124
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 6
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;freebsd.org.       IN   A

;; ANSWER SECTION:
freebsd.org.     3600   IN   A   8.8.178.110

;; AUTHORITY SECTION:
freebsd.org.     3600   IN   NS   ns1.isc-sns.net.
freebsd.org.     3600   IN   NS   ns3.isc-sns.info.
freebsd.org.     3600   IN   NS   ns2.isc-sns.com.

;; ADDITIONAL SECTION:
ns1.isc-sns.net.   3600   IN   A   72.52.71.1
ns1.isc-sns.net.   3600   IN   AAAA   2001:470:1a::1
ns2.isc-sns.com.   3600   IN   A   38.103.2.1
ns3.isc-sns.info.   3600   IN   A   63.243.194.1
ns3.isc-sns.info.   3600   IN   AAAA   2001:5a0:10::1

;; Query time: 32 msec
;; SERVER: 72.52.71.1#53(72.52.71.1)
;; WHEN: Thu Oct 30 13:58:37 EDT 2014
;; MSG SIZE  rcvd: 248
I flushed DNS, restarted DNS, rebooted the system, all to no avail. Next I effectively disabled my firewall with ipfw add 1 allow ip any any and it started working. I made no changes to my firewall rules in the months prior to this (which on my workstations is simply set[]up with
Code:
firewall_type="workstation"
in rc.conf) so am not sure what changed.
Code:
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
01100 check-state
01200 allow tcp from me to any established
01300 allow tcp from me to any setup keep-state
01400 allow udp from me to any keep-state
01500 allow icmp from me to any keep-state
01600 allow ipv6-icmp from me to any keep-state
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 dst-port 67 out
01800 allow udp from any 67 to me dst-port 68 in
01900 allow udp from any 67 to 255.255.255.255 dst-port 68 in
02000 allow udp from fe80::/10 to me dst-port 546 in
02100 allow icmp from any to any icmptypes 8
02200 allow ipv6-icmp from any to any ip6 icmp6types 128,129
02300 allow icmp from any to any icmptypes 3,4,11
02400 allow ipv6-icmp from any to any ip6 icmp6types 3
02500 allow ip from 10.0.201.0/24 to me
02600 allow ip from <REMOVED>/27 to me
65000 count ip from any to any
65100 deny { tcp or udp } from any to any dst-port 135-139,445 in
65200 deny { tcp or udp } from any to any dst-port 1026,1027 in
65300 deny { tcp or udp } from any to any dst-port 1433,1434 in
65400 deny ip from any to 255.255.255.255
65500 deny ip from any to 224.0.0.0/24 in
65500 deny udp from any to any dst-port 520 in
65500 deny tcp from any 80,443 to any dst-port 1024-65535 in
65500 deny ip from any to any
65535 deny ip from any to any
I then narrowed it down to the final ipfw rule by deleting the
Code:
1 allow ip any any
rule and inserting
Code:
allow udp from any to any
right before the
Code:
65500 deny ip from any to any
rule. At this point I'm thinking something broke in the UDP keep-state, but don't know why it would effect all four of my systems. I've have a FreeBSD 10.0-RELEASE-p11 box at work with the exact same initial firewall rules and it still resolves ( dig +trace) freebsd.org fine on it. Any ideas?
 
Interesting addition: today I loaded a new box with 9.3-RELEASE at work and applied the workstation firewall rules to it and it dies at the exact same point in a dig +trace.
 
If you think one of your deny rules in the set might be blocking inbound traffic, then it might help to add the "log" keyword to those rules. For me, the logging appends records to /var/log/security, up to the "loglimit" keyword per rule (or default). The logfile entry includes firewall ruleset number, action, protocol, IP address and port, and interface and direction. Example:
Code:
Nov  2 05:13:18 myhostname kernel: ipfw: 139xxx Deny UDP a.b.c.d:33995 w.x.y.z:33434 in via bge0
I've personally found selective logging invaluable in helping ensure my ruleset was in reality doing what I intended.
 
I guess my point is I don't think it's my deny rule, once again this is the vanilla "workstation" firewall type in the /etc/rc.conf which I have used for years now and which I have now verified on multiple fresh 9.3-RELEASE systems (before and after system patches) on different networks (different class C networks and with and without dual stack IPv6 enabled). These exact same rules on a 10.0-RELEASE box do not produce this failure to resolve the *.freebsd.org domains. Logging the final deny rule doesn't give me anything more to go on, maybe it will for someone else:
Code:
Nov  4 12:27:34 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 25360:8@1472+)
Nov  4 12:27:34 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 25360:231@1480)
Nov  4 12:27:35 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 45673:8@1472+)
Nov  4 12:27:35 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 45673:231@1480)
Nov  4 12:27:36 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 59524:8@1472+)
Nov  4 12:27:36 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 59524:231@1480)
Nov  4 12:27:41 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 43965:8@1472+)
Nov  4 12:27:41 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 43965:231@1480)
Nov  4 12:27:42 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 52524:8@1472+)
Nov  4 12:27:42 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 52524:231@1480)
Nov  4 12:27:43 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 6041:8@1472+)
Nov  4 12:27:43 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 6041:231@1480)
Nov  4 12:27:48 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 62896:8@1472+)
Nov  4 12:27:48 FBSD9 kernel: ipfw: 65520 Deny UDP 72.52.71.1 98.158.211.3 in via em0 (frag 62896:231@1480)
Nov  4 12:27:49 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 59677:8@1472+)
Nov  4 12:27:49 FBSD9 kernel: ipfw: 65520 Deny UDP 38.103.2.1 98.158.211.3 in via em0 (frag 59677:231@1480)
Nov  4 12:27:51 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 18747:8@1472+)
Nov  4 12:27:51 FBSD9 kernel: ipfw: 65520 Deny UDP 63.243.194.1 98.158.211.3 in via em0 (frag 18747:231@1480)
In my testing today I've now narrowed it down to the DNSSEC on the domain. So while a dig +trace freebsd.org fails, a dig +trace +nodnssec freebsd.org fully resolves.
Code:
{53}root@FreeBSD9(~)> dig +trace freebsd.org

; <<>> DiG 9.9.5 <<>> +trace freebsd.org
;; global options: +cmd
.       89292   IN   NS   k.root-servers.net.
.       89292   IN   NS   a.root-servers.net.
.       89292   IN   NS   g.root-servers.net.
.       89292   IN   NS   i.root-servers.net.
.       89292   IN   NS   d.root-servers.net.
.       89292   IN   NS   c.root-servers.net.
.       89292   IN   NS   e.root-servers.net.
.       89292   IN   NS   f.root-servers.net.
.       89292   IN   NS   b.root-servers.net.
.       89292   IN   NS   l.root-servers.net.
.       89292   IN   NS   m.root-servers.net.
.       89292   IN   NS   j.root-servers.net.
.       89292   IN   NS   h.root-servers.net.
.       501389   IN   RRSIG   NS 8 0 518400 20141111050000 20141104040000 22603 . Mvyx+yHe42PTV96noLMs8NpHkO0JEMj2PyEsfIh0jw0r2+XwWiW2kT6T wUOJ3Dp3jzwEub7AkFrpkSU9xDQYPwVbIUT9JcA2kQ1KKZzQ2P0Uv15b Ai7bp823wJGUURhgdM9u1b9QrwOuI17ldUI2U0uDbfr3HBil8xGQzFII r8Y=
;; Received 913 bytes from 206.162.192.2#53(206.162.192.2) in 42 ms

org.       172800   IN   NS   d0.org.afilias-nst.org.
org.       172800   IN   NS   b0.org.afilias-nst.org.
org.       172800   IN   NS   b2.org.afilias-nst.org.
org.       172800   IN   NS   a2.org.afilias-nst.info.
org.       172800   IN   NS   a0.org.afilias-nst.info.
org.       172800   IN   NS   c0.org.afilias-nst.info.
org.       86400   IN   DS   21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0 D90F01BA
org.       86400   IN   DS   21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org.       86400   IN   RRSIG   DS 8 1 86400 20141111170000 20141104160000 22603 . fIdiT4FawjFcLbyCpkwb5pd/jraDEYYrhGtGKoUOiuo2CSEbPZktjMWq w5lDdBsx1PVdAeh24G307knkb4B0YYJAJVhMBJs8DCiFzyuhmSMosaxx WI5jxFqgb9nOgMV0Ktiab9Ou2DNlZ4BP1sjy1Um7U4726njQqoN7dwkB StM=
;; Received 685 bytes from 192.203.230.10#53(e.root-servers.net) in 150 ms

freebsd.org.     86400   IN   NS   ns1.isc-sns.net.
freebsd.org.     86400   IN   NS   ns2.isc-sns.com.
freebsd.org.     86400   IN   NS   ns3.isc-sns.info.
freebsd.org.     86400   IN   DS   32659 8 2 AF3B32E46DF2FC32C0110C7D6B808EE73E0411501AFAF9022D3DCD0A FA5B3ACD
freebsd.org.     86400   IN   RRSIG   DS 7 2 86400 20141122155450 20141101145450 60764 org. DB4Plbzlv9qzmWgGQSYnlFrJZwOeAFRLVXHjCB3aDLuT9mI9C/HiGwiK tDj5sAU6UCR31rRArLTWz4CaZfPTqTdjXoNoXfP8Jy6l/KjVBJsK5Oim wPAkIrKVj3BKZzcTmFnoEVzK4zk6eYOs8XFTGpDHSCWSCsRrFKNuLrpS n9E=
;; Received 339 bytes from 199.249.112.1#53(a2.org.afilias-nst.info) in 81 ms

;; connection timed out; no servers could be reached
{54}root@FreeBSD9(~)> dig +trace +nodnssec freebsd.org

; <<>> DiG 9.9.5 <<>> +trace +nodnssec freebsd.org
;; global options: +cmd
.       89264   IN   NS   e.root-servers.net.
.       89264   IN   NS   g.root-servers.net.
.       89264   IN   NS   a.root-servers.net.
.       89264   IN   NS   m.root-servers.net.
.       89264   IN   NS   b.root-servers.net.
.       89264   IN   NS   j.root-servers.net.
.       89264   IN   NS   h.root-servers.net.
.       89264   IN   NS   f.root-servers.net.
.       89264   IN   NS   l.root-servers.net.
.       89264   IN   NS   i.root-servers.net.
.       89264   IN   NS   d.root-servers.net.
.       89264   IN   NS   k.root-servers.net.
.       89264   IN   NS   c.root-servers.net.
;; Received 755 bytes from 206.162.192.2#53(206.162.192.2) in 44 ms

org.       172800   IN   NS   d0.org.afilias-nst.org.
org.       172800   IN   NS   c0.org.afilias-nst.info.
org.       172800   IN   NS   a0.org.afilias-nst.info.
org.       172800   IN   NS   b0.org.afilias-nst.org.
org.       172800   IN   NS   b2.org.afilias-nst.org.
org.       172800   IN   NS   a2.org.afilias-nst.info.
;; Received 442 bytes from 192.36.148.17#53(i.root-servers.net) in 127 ms

freebsd.org.     86400   IN   NS   ns1.isc-sns.net.
freebsd.org.     86400   IN   NS   ns3.isc-sns.info.
freebsd.org.     86400   IN   NS   ns2.isc-sns.com.
;; Received 128 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 43 ms

freebsd.org.     3600   IN   A   8.8.178.110
freebsd.org.     3600   IN   NS   ns1.isc-sns.net.
freebsd.org.     3600   IN   NS   ns3.isc-sns.info.
freebsd.org.     3600   IN   NS   ns2.isc-sns.com.
;; Received 248 bytes from 72.52.71.1#53(ns1.isc-sns.net) in 36 ms
At this point I'm feeling like something changed with freebsd.org's DNSSEC on or around 10/29/14 which FreeBSD 9.3 itself doesn't like. Personally I can deal, but it just seems kind of silly that a FreeBSD 9.3 box's own firewall prevents access to the FreeBSD.org domains (without the aid of external unhindered nameservers) yet blocks no other domains that I've come across yet. Then again, maybe I'm the only person who runs their own internal nameserver for home use?
 
Interestingly enough there was another post on here recently with a very similar problem. They tracked it down to DNSSEC packets being fragmented and blocked by their pf firewall. The frag keyword in your log output suggests you may also be seeing fragmented packets and possibly the same problem.

They fixed their problem by using scrub on interface to have packets reassembled before being checked by the firewall. I don't know if there's a similar function in ipfw?
 
Good catch usdmatt, you nailed it. One can use
Code:
ipfw add pass udp from any to any frag
or
Code:
ipfw add reass udp from any to any in
to ignore or fix the fragmenting. I tried the latter which did in fact enable the DNS query to complete.

I guess my thought now is, should this not be added to all /etc/rc.firewall types moving forward so it doesn't randomly catch others off guard in the future?
 
Sorry that I am digging out this old thread, but since I just came across this. Has the inclusion into rc.firewall been discussed somewhere yet? Would it make sense to open a PR for that?
 
I only observed this with 9.3, which was EoL'ed in January, so I'm not sure it's an issue at this point.
 
Interesting, I had since (heavily) changed the IPFW rules on my primary box so hadn't seen it when it went to 11-REL. For grins, I installed bind910-9.10.4P5_1 on another 11.0-RELEASE-p7 box using the "workstation" firewall and I do see this issue again. That said, I'm unsure if a PR is appropriate or not since BIND isn't in base anymore and as an added service it seems reasonable to me that one might have to tweak IPFW rules for it? Don't let me discourage you though, if you think it warrants one, go for it.
 
Unbound (local_unbound service) is in base now and causes this too (after using service local_unbound setup, etc.).
 
Back
Top