As of yesterday afternoon I stopped receiving emails from the freebsd.org mailing lists on my personal domain/network and upon further digging today found that I was unable to access freebsd.org or any of its sub-domains. Near as I can tell I had/have no problems with any other domains and after "digging" into this for freebsd.org I found the problem to be DNS related. I simply changed my primary DNS to an external system and all works. Problem averted, but I still want to know why it stopped working and how I can fix it internally.
So, let's start with what I've determined. First off, this is happening to all four of my FreeBSD 9.3-RELEASE-p4 systems on my personal network that are dual stacked with public IPv4 and IPv6 addresses. When I do a
Yet a direct query against those last nameservers does work?
I flushed DNS, restarted DNS, rebooted the system, all to no avail. Next I effectively disabled my firewall with
in rc.conf) so am not sure what changed.
I then narrowed it down to the final ipfw rule by deleting the
rule and inserting
right before the
rule. At this point I'm thinking something broke in the UDP keep-state, but don't know why it would effect all four of my systems. I've have a FreeBSD 10.0-RELEASE-p11 box at work with the exact same initial firewall rules and it still resolves (
So, let's start with what I've determined. First off, this is happening to all four of my FreeBSD 9.3-RELEASE-p4 systems on my personal network that are dual stacked with public IPv4 and IPv6 addresses. When I do a
dig +trace freebsd.org from my network, I get through all the nameservers until the final A record, which times out (AAAA also times out):
Code:
{17}root@hydra(~)> dig +trace freebsd.org
; <<>> DiG 9.9.5 <<>> +trace freebsd.org
;; global options: +cmd
. 3600000 IN NS M.ROOT-SERVERS.NET.
. 3600000 IN NS K.ROOT-SERVERS.NET.
. 3600000 IN NS B.ROOT-SERVERS.NET.
. 3600000 IN NS E.ROOT-SERVERS.NET.
. 3600000 IN NS J.ROOT-SERVERS.NET.
. 3600000 IN NS H.ROOT-SERVERS.NET.
. 3600000 IN NS D.ROOT-SERVERS.NET.
. 3600000 IN NS F.ROOT-SERVERS.NET.
. 3600000 IN NS A.ROOT-SERVERS.NET.
. 3600000 IN NS C.ROOT-SERVERS.NET.
. 3600000 IN NS I.ROOT-SERVERS.NET.
. 3600000 IN NS G.ROOT-SERVERS.NET.
. 3600000 IN NS L.ROOT-SERVERS.NET.
;; Received 797 bytes from 206.162.192.2#53(206.162.192.2) in 54 ms
org. 172800 IN NS a2.org.afilias-nst.info.
org. 172800 IN NS d0.org.afilias-nst.org.
org. 172800 IN NS a0.org.afilias-nst.info.
org. 172800 IN NS b2.org.afilias-nst.org.
org. 172800 IN NS b0.org.afilias-nst.org.
org. 172800 IN NS c0.org.afilias-nst.info.
org. 86400 IN DS 21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org. 86400 IN DS 21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0 D90F01BA
org. 86400 IN RRSIG DS 8 1 86400 20141106050000 20141030040000 22603 . BNPULda5SeKowJUHPxSXRPv9Cwc/l/u3iGD8cfg1YrF71uIxOzBkPYpf qeHXIPCkBMAGGMNPyjMAv/sdF7eykDVSGC1swerDdF+2qRUrA7lM6wc8 Tud/4Vk3Q80V4lAxnLW2ApfnXOaljkrbVBlAzwVC3D30tRgL4S9obQAU BRY=
;; Received 685 bytes from 192.228.79.201#53(B.ROOT-SERVERS.NET) in 95 ms
freebsd.org. 86400 IN NS ns2.isc-sns.com.
freebsd.org. 86400 IN NS ns1.isc-sns.net.
freebsd.org. 86400 IN NS ns3.isc-sns.info.
freebsd.org. 86400 IN DS 32659 8 2 AF3B32E46DF2FC32C0110C7D6B808EE73E0411501AFAF9022D3DCD0A FA5B3ACD
freebsd.org. 86400 IN RRSIG DS 7 2 86400 20141115155009 20141025145009 57479 org. SDAo+3o5ILkTI1fKUmQW/EsOU3nNKiDlWcxweiX5ND8OLABiy7Bx6YYd KknL6dEQB7NJq50RELLQY5JfbXsf5ormH9bgswOHjS+bVbosSRTiHdnx Mcp20/H6Tkw8V0VXkHhSGvbZ+Zh2m30NGv7PjM1nTbiSINWutgvRY2CR 0wQ=
;; Received 339 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 45 ms
;; connection timed out; no servers could be reachedYet a direct query against those last nameservers does work?
Code:
{33}root@hydra(~)> dig freebsd.org @ns1.isc-sns.net
; <<>> DiG 9.9.5 <<>> freebsd.org @ns1.isc-sns.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56124
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 6
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;freebsd.org. IN A
;; ANSWER SECTION:
freebsd.org. 3600 IN A 8.8.178.110
;; AUTHORITY SECTION:
freebsd.org. 3600 IN NS ns1.isc-sns.net.
freebsd.org. 3600 IN NS ns3.isc-sns.info.
freebsd.org. 3600 IN NS ns2.isc-sns.com.
;; ADDITIONAL SECTION:
ns1.isc-sns.net. 3600 IN A 72.52.71.1
ns1.isc-sns.net. 3600 IN AAAA 2001:470:1a::1
ns2.isc-sns.com. 3600 IN A 38.103.2.1
ns3.isc-sns.info. 3600 IN A 63.243.194.1
ns3.isc-sns.info. 3600 IN AAAA 2001:5a0:10::1
;; Query time: 32 msec
;; SERVER: 72.52.71.1#53(72.52.71.1)
;; WHEN: Thu Oct 30 13:58:37 EDT 2014
;; MSG SIZE rcvd: 248 ipfw add 1 allow ip any any and it started working. I made no changes to my firewall rules in the months prior to this (which on my workstations is simply set[]up with
Code:
firewall_type="workstation"
Code:
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
01100 check-state
01200 allow tcp from me to any established
01300 allow tcp from me to any setup keep-state
01400 allow udp from me to any keep-state
01500 allow icmp from me to any keep-state
01600 allow ipv6-icmp from me to any keep-state
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 dst-port 67 out
01800 allow udp from any 67 to me dst-port 68 in
01900 allow udp from any 67 to 255.255.255.255 dst-port 68 in
02000 allow udp from fe80::/10 to me dst-port 546 in
02100 allow icmp from any to any icmptypes 8
02200 allow ipv6-icmp from any to any ip6 icmp6types 128,129
02300 allow icmp from any to any icmptypes 3,4,11
02400 allow ipv6-icmp from any to any ip6 icmp6types 3
02500 allow ip from 10.0.201.0/24 to me
02600 allow ip from <REMOVED>/27 to me
65000 count ip from any to any
65100 deny { tcp or udp } from any to any dst-port 135-139,445 in
65200 deny { tcp or udp } from any to any dst-port 1026,1027 in
65300 deny { tcp or udp } from any to any dst-port 1433,1434 in
65400 deny ip from any to 255.255.255.255
65500 deny ip from any to 224.0.0.0/24 in
65500 deny udp from any to any dst-port 520 in
65500 deny tcp from any 80,443 to any dst-port 1024-65535 in
65500 deny ip from any to any
65535 deny ip from any to any
Code:
1 allow ip any any
Code:
allow udp from any to any
Code:
65500 deny ip from any to any dig +trace) freebsd.org fine on it. Any ideas?