BIND dig takes ~2000-4000+ msec from internal but >30msec from Internet DMZ

In the last 6 weeks I have gone from limited knowledge of FreeBSD and BIND DNS (only knew it existed) to asking questions in house that no one knows the answer to about FreeBSD and BIND. So I turn to this forum with my questions. Please excuse any details/info which appears vague or censored as it is a mandatory requirement of the environment.


Question:
I do a dig from my internal BIND DNS server for an Internet FQDN and the dig comes back in ~2000-4000+ msec or it just does not resolve. However if I do a dig from the Internet DNS server, it takes ~5-30 msec to resolve. The firewall for Internet and Closed_Network are same brand and type. DIGs from INTRANET_BIND toward the Closed_Network_BIND result in responses in the range of <1-10 msec. I am unsure where else to look in my config or if I have a misconfiguration somewhere. Any help is welcome and appreciated.

* ICMP is not allowed between the server segment and the Internet DMZ for troubleshooting


Environment brief:
Summary:
Attempting to move toward having all intranet systems point to INTRANET_BIND instead of current DNS solution. Existing solution has Forwarders for INTERNET_BIND_01/02 and Remote_DNS_01/02/03 while also having conditional forwarders for Closed_Network_BIND_01/02. There is a severe difference using the existing solution (fast) and the future solution (slow and spotty) in speed. End/future solution should hopefully handle ~20-24 million queries a day.


Intranet BIND DNS
* Configured as a Forwarder/Cache server
* Forwards internal FQDN to an internal DNS
* Forwards default/Internet DNS queries to the Internet BIND DNS servers
* Forwards specific FQDNs to Closed_Network BIND DNS servers for custom resolution
* /etc/resolv.conf has itself, Internet DNS #1, and Internet DNS #2
* root hints disabled within named.conf
Internet BIND DNS
* Primary and Secondary both configured as Forwarder/Cache
* Forwards everything to ISP's DNS servers
* /etc/resolv.conf has ISP DNS #1 and ISP DNS #2
* root hints enabled within named.conf
Closed_Network BIND DNS
* Primary and secondary both configured as Forwarder/Cache
* Forwards everything to remote DNS servers on closed network
* /etc/resolv.conf has itself, Remote_DNS_01, and Remote_DNS_02
* root hints disabled within named.conf
* Traffic goes out separate circuit from the Internet

Example:
Code:
[u]INTRANET_BIND[/u]# dig i.dell.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> i.dell.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63153
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 13, ADDITIONAL: 3

;; QUESTION SECTION:
;i.dell.com.                    IN      A

;; ANSWER SECTION:
i.dell.com.             0       IN      CNAME   img.dell-cidr.akadns.net.
img.dell-cidr.akadns.net. 0     IN      CNAME   ccdn-global.dell.com.edgesuite.net.
ccdn-global.dell.com.edgesuite.net. 0 IN CNAME  ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net.
ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net. 0 IN CNAME a1058.g.akamai.net.
a1058.g.akamai.net.     6       IN      A       96.17.75.32
a1058.g.akamai.net.     6       IN      A       96.17.75.34

;; AUTHORITY SECTION:
.                       7786    IN      NS      i.root-servers.net.
.                       7786    IN      NS      j.root-servers.net.
.                       7786    IN      NS      c.root-servers.net.
.                       7786    IN      NS      h.root-servers.net.
.                       7786    IN      NS      e.root-servers.net.
.                       7786    IN      NS      f.root-servers.net.
.                       7786    IN      NS      a.root-servers.net.
.                       7786    IN      NS      g.root-servers.net.
.                       7786    IN      NS      d.root-servers.net.
.                       7786    IN      NS      l.root-servers.net.
.                       7786    IN      NS      k.root-servers.net.
.                       7786    IN      NS      m.root-servers.net.
.                       7786    IN      NS      b.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     247403  IN      A       198.41.0.4
a.root-servers.net.     247734  IN      AAAA    2001:503:ba3e::2:30
c.root-servers.net.     318287  IN      A       192.33.4.12

;; Query time: 3160 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May  9 11:07:56 2011
;; MSG SIZE  rcvd: 501
Code:
[u]INTERNET_BIND_01[/u]# dig i.dell.com @68.87.74.162

; <<>> DiG 9.6.-ESV-R3 <<>> i.dell.com @68.87.74.162
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15209
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;i.dell.com.                    IN      A

;; ANSWER SECTION:
i.dell.com.             0       IN      CNAME   img.dell-cidr.akadns.net.
img.dell-cidr.akadns.net. 0     IN      CNAME   ccdn-global.dell.com.edgesuite.net.
ccdn-global.dell.com.edgesuite.net. 0 IN CNAME  ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net.
ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net. 0 IN CNAME a1058.g.akamai.net.
a1058.g.akamai.net.     20      IN      A       96.17.146.43
a1058.g.akamai.net.     20      IN      A       96.17.146.75

;; Query time: 42 msec
;; SERVER: 68.87.74.162#53(68.87.74.162)
;; WHEN: Mon May  9 11:14:22 2011
;; MSG SIZE  rcvd: 233

Did a tcpdump on both servers while doing another dig with the following results:
Code:
INTRANET_BIND# dig i.dell.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> i.dell.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54648
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 13, ADDITIONAL: 3

;; QUESTION SECTION:
;i.dell.com.                    IN      A

;; ANSWER SECTION:
i.dell.com.             0       IN      CNAME   img.dell-cidr.akadns.net.
img.dell-cidr.akadns.net. 0     IN      CNAME   ccdn-global.dell.com.edgesuite.net.
ccdn-global.dell.com.edgesuite.net. 0 IN CNAME  ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net.
ccdn-global.dell.com.edgesuite.net.globalredir.akadns.net. 0 IN CNAME a1058.g.akamai.net.
a1058.g.akamai.net.     5       IN      A       96.17.146.75
a1058.g.akamai.net.     5       IN      A       96.17.146.43

;; AUTHORITY SECTION:
.                       6777    IN      NS      e.root-servers.net.
.                       6777    IN      NS      k.root-servers.net.
.                       6777    IN      NS      m.root-servers.net.
.                       6777    IN      NS      g.root-servers.net.
.                       6777    IN      NS      a.root-servers.net.
.                       6777    IN      NS      j.root-servers.net.
.                       6777    IN      NS      l.root-servers.net.
.                       6777    IN      NS      d.root-servers.net.
.                       6777    IN      NS      i.root-servers.net.
.                       6777    IN      NS      h.root-servers.net.
.                       6777    IN      NS      f.root-servers.net.
.                       6777    IN      NS      c.root-servers.net.
.                       6777    IN      NS      b.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     246394  IN      A       198.41.0.4
a.root-servers.net.     246725  IN      AAAA    2001:503:ba3e::2:30
c.root-servers.net.     317278  IN      A       192.33.4.12

;; Query time: 511 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May  9 11:24:45 2011
;; MSG SIZE  rcvd: 501
Code:
INTRANET_BIND# tcpdump -vv host 10.XXX.XXX.153
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
11:24:35.137716 IP (tos 0x0, ttl 64, id 62859, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [bad udp cksum fcbc!] 54648+ A? i.dell.com. (28)
11:24:40.138611 IP (tos 0x0, ttl 64, id 62865, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [bad udp cksum fcbc!] 54648+ A? i.dell.com. (28)
11:24:45.147943 IP (tos 0x0, ttl 64, id 62869, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [bad udp cksum fcbc!] 54648+ A? i.dell.com. (28)
11:24:45.658740 IP (tos 0x0, ttl 62, id 60507, offset 0, flags [none], proto UDP (17), length 529)
    INTERNET_BIND_01.domain > INTRANET_BIND.36467: 54648 q: A? i.dell.com. 6/13/3 i.dell.com. CNAME[|domain]
Code:
INTERNET_BIND_01# tcpdump -vv host 10.X.X.53
tcpdump: listening on bce0, link-type EN10MB (Ethernet), capture size 96 bytes
11:24:18.347858 IP (tos 0x0, ttl 62, id 62859, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [udp sum ok] 54648+ A? i.dell.com. (28)
11:24:23.348704 IP (tos 0x0, ttl 62, id 62865, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [udp sum ok] 54648+ A? i.dell.com. (28)
11:24:28.358008 IP (tos 0x0, ttl 62, id 62869, offset 0, flags [none], proto UDP (17), length 56)
    INTRANET_BIND.36467 > INTERNET_BIND_01.domain: [udp sum ok] 54648+ A? i.dell.com. (28)
11:24:28.868442 IP (tos 0x0, ttl 64, id 60507, offset 0, flags [none], proto UDP (17), length 529)
    INTERNET_BIND_01.domain > INTRANET_BIND.36467: 54648 q: A? i.dell.com. 6/13/3 i.dell.com. CNAME[|domain]
11:24:33.446556 IP (tos 0x0, ttl 64, id 61464, offset 0, flags [none], proto UDP (17), length 529)
    INTERNET_BIND_01.domain > INTRANET_BIND.36467: 54648 q: A? i.dell.com. 6/13/3 i.dell.com. CNAME[|domain]
 
w3nd377 said:
* ICMP is not allowed between the server segment and the Internet DMZ for troubleshooting
That is too bad not only for troubleshooting but for possible fragmentation issues.

Start by capturing the traffic from its original source to the destination (internet). Eliminate first any network issues at all respective layers and then move on to the application.

Regards,
George
 
George,

Was able to obtain the following information regarding ICMP data from a server on same subnet as INTRANET_BIND_DNS and from a network workstation.

From Server on same network as INTRANET_BIND_DNS
Code:
C:\Users\ELMO>tracert 10.XXX.XXX.154

Tracing route to INTERNET_BIND_DNS_02 [10.XXX.XXX.154]
over a maximum of 30 hops:

  1    26 ms    <1 ms     3 ms  10.X.X.2
  2    <1 ms    <1 ms    <1 ms  10.XX.XX.1
  3    <1 ms    <1 ms    <1 ms  INTERNET_BIND_DNS [10.XXX.XXX.154]

Trace complete.

C:\Users\ELMO>tracert 10.X.XXX.153

Tracing route to CLOSED_NETWORK_BIND_DNS_01 [10.X.XXX.153]
over a maximum of 30 hops:

  1    11 ms    <1 ms    <1 ms  10.X.X.2
  2    <1 ms    <1 ms    <1 ms  10.X.XXX.17
  3    <1 ms    <1 ms    <1 ms  CLOSED_NETWORK_BIND_DNS [10.X.XXX.153]

Trace complete.


C:\Users\ELMO>ping 10.XXX.XXX.154

Pinging 10.XXX.XXX.154 with 32 bytes of data:
Reply from 10.XXX.XXX.154: bytes=32 time=12ms TTL=62
Reply from 10.XXX.XXX.154: bytes=32 time<1ms TTL=62
Reply from 10.XXX.XXX.154: bytes=32 time<1ms TTL=62
Reply from 10.XXX.XXX.154: bytes=32 time<1ms TTL=62

Ping statistics for 10.XXX.XXX.154:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 12ms, Average = 3ms

From local system to INTERNET_BIND_DNS_02
Code:
C:\Users\ELMO>pathping -4 10.XXX.XXX.154

Tracing route to INTERNET_BIND_DNS_02 [10.XXX.XXX.154]
over a maximum of 30 hops:
  0  localhost [10.X.XXX.99]
  1  10.X.XXX.2
  2  10.XX.XX.1
  3  INTERNET_BIND_DNS_02 [10.XXX.XXX.154]

Computing statistics for 75 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           localhost [10.X.XXX.99] 
                                0/ 100 =  0%   |
  1    3ms     0/ 100 =  0%     0/ 100 =  0%  10.X.XXX.2
                                0/ 100 =  0%   |
  2    2ms     0/ 100 =  0%     0/ 100 =  0%  10.XX.XX.1
                                0/ 100 =  0%   |
  3    0ms     0/ 100 =  0%     0/ 100 =  0%  INTERNET_BIND_DNS_02 [10.XXX.XXX.154]

Trace complete.
 
@w3nd377,

Machines that are located in the same network can not be firewalled.

Now, I don't know your physical network topology to suggest something more intelligent. But if I was in your shoes I would use a sniffer like wireshark to capture a full query from a station with the problem and follow the full path until I get a reply. That way you will know which DNS is asking who and who is slowing you down. This might not be a network issue but the sniffer can give you some clues on where to look.
 
@gkontos

So evidently the network SME has taken a look at the situation and he is pointing the finger at BIND DNS. I have compiled some debug 3 level logs within BIND along with TCPDUMP from both sides. I failed to get a TCPDUMP of the traffic leaving the forwarder into the Internet but could do that on next attempt to capture info if necessary. (submitted in parts to be within 10000 character limit)

Unfortunately, I have now created several more questions instead of answering the existing ones.
1) Why the differences in time when repeatedly requesting the same info?
2) Why the extreme difference between asking the local BIND service (~4000 msec) and asking the ISP BIND service (17 & 39 msec)?
3) Is letting the BIND cache be as large as it wants creating more issues?

Debug Level 3 Output from BIND:

Code:
...
16-May-2011 14:30:02.070 queries: info: client 10.X.X.53#45245: query: www.cdwg.com IN A +
16-May-2011 14:30:02.070 security: debug 3: client 10.X.X.53#45245: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:02.070 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:02.070 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): create
16-May-2011 14:30:02.070 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): join
16-May-2011 14:30:02.070 resolver: debug 3: fetch 0x80477d350 (fctx 0x8032b0000(www.cdwg.com/A)): created
16-May-2011 14:30:02.071 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): start
16-May-2011 14:30:02.071 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): try
16-May-2011 14:30:02.071 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:02.071 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:02.071 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): query
16-May-2011 14:30:02.071 resolver: debug 3: resquery 0x8032a8840 (fctx 0x8032b0000(www.cdwg.com/A)): send
16-May-2011 14:30:02.071 resolver: debug 3: resquery 0x8032a8840 (fctx 0x8032b0000(www.cdwg.com/A)): sent
16-May-2011 14:30:02.071 resolver: debug 3: resquery 0x8032a8840 (fctx 0x8032b0000(www.cdwg.com/A)): senddone
16-May-2011 14:30:02.104 resolver: debug 3: resquery 0x8032a8840 (fctx 0x8032b0000(www.cdwg.com/A)): response
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): answer_response
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): cache_message
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): clone_results
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): done
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): sendevents
16-May-2011 14:30:02.104 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): create
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): join
16-May-2011 14:30:02.104 resolver: debug 3: fetch 0x8010849d0 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): created
16-May-2011 14:30:02.104 resolver: debug 3: fetch 0x80477d350 (fctx 0x8032b0000(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): shutdown
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): start
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x8032b0000(www.cdwg.com/A'): destroy
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:02.104 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:02.104 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:02.104 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:02.104 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:04.105 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:04.105 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:04.105 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:04.105 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:04.105 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:04.105 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:04.105 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:06.107 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:06.107 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:06.107 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:06.107 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:06.166 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): response
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): answer_response
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): noanswer_response
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cache_message
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): clone_results
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): done
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): sendevents
16-May-2011 14:30:06.166 resolver: debug 3: fetch 0x8010849d0 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): destroyfetch
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): shutdown
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): doshutdown
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:06.166 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): destroy
 
Code:
16-May-2011 14:30:07.078 queries: info: client 10.X.X.53#45245: query: www.cdwg.com IN A +
16-May-2011 14:30:07.078 security: debug 3: client 10.X.X.53#45245: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:07.078 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): create
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): join
16-May-2011 14:30:07.078 resolver: debug 3: fetch 0x80477d730 (fctx 0x803284000(www.cdwg.com/A)): created
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): start
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): try
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:07.078 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): query
16-May-2011 14:30:07.078 resolver: debug 3: resquery 0x8032a8b00 (fctx 0x803284000(www.cdwg.com/A)): send
16-May-2011 14:30:07.078 resolver: debug 3: resquery 0x8032a8b00 (fctx 0x803284000(www.cdwg.com/A)): sent
16-May-2011 14:30:07.078 resolver: debug 3: resquery 0x8032a8b00 (fctx 0x803284000(www.cdwg.com/A)): senddone
16-May-2011 14:30:07.205 resolver: debug 3: resquery 0x8032a8b00 (fctx 0x803284000(www.cdwg.com/A)): response
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): answer_response
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): cache_message
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): clone_results
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): done
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): sendevents
16-May-2011 14:30:07.205 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): create
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): join
16-May-2011 14:30:07.205 resolver: debug 3: fetch 0x80477d390 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): created
16-May-2011 14:30:07.205 resolver: debug 3: fetch 0x80477d730 (fctx 0x803284000(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): shutdown
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): start
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x803284000(www.cdwg.com/A'): destroy
16-May-2011 14:30:07.205 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:07.205 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:07.205 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:07.205 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:09.206 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:09.206 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:09.206 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:09.206 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:09.206 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:09.206 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:09.206 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:09.246 resolver: debug 3: resquery 0x80340b840 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): response
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): answer_response
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): noanswer_response
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cache_message
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): clone_results
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): done
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): sendevents
16-May-2011 14:30:09.246 resolver: debug 3: fetch 0x80477d390 (fctx 0x80342a000(www.cdwg.com.edgekey.net/A)): destroyfetch
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): shutdown
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): doshutdown
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:09.246 resolver: debug 3: fctx 0x80342a000(www.cdwg.com.edgekey.net/A'): destroy
 
Code:
16-May-2011 14:30:09.702 queries: info: client 10.X.X.53#44228: query: www.cdwg.com IN A +
16-May-2011 14:30:09.702 security: debug 3: client 10.X.X.53#44228: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:09.702 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:09.702 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): create
16-May-2011 14:30:09.702 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): join
16-May-2011 14:30:09.702 resolver: debug 3: fetch 0x80477d200 (fctx 0x80329b400(www.cdwg.com/A)): created
16-May-2011 14:30:09.702 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): start
16-May-2011 14:30:09.702 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): try
16-May-2011 14:30:09.702 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:09.703 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:09.703 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): query
16-May-2011 14:30:09.703 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): send
16-May-2011 14:30:09.703 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): sent
16-May-2011 14:30:09.703 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): senddone
16-May-2011 14:30:09.741 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): response
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): answer_response
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cache_message
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): clone_results
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): done
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): sendevents
16-May-2011 14:30:09.741 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): create
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): join
16-May-2011 14:30:09.741 resolver: debug 3: fetch 0x80477d5c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): created
16-May-2011 14:30:09.741 resolver: debug 3: fetch 0x80477d200 (fctx 0x80329b400(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): shutdown
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): start
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): destroy
16-May-2011 14:30:09.741 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:09.742 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:09.742 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:09.742 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:09.742 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:09.742 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:09.742 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:11.743 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:11.743 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:11.743 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:11.743 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:11.743 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:11.743 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:11.743 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:13.744 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:13.744 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:13.744 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:13.744 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:13.744 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:13.745 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:13.745 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:13.745 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:13.745 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): senddone
 
Code:
16-May-2011 14:30:14.707 queries: info: client 10.X.X.53#44228: query: www.cdwg.com IN A +
16-May-2011 14:30:14.707 security: debug 3: client 10.X.X.53#44228: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:14.707 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): create
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): join
16-May-2011 14:30:14.707 resolver: debug 3: fetch 0x80477dc40 (fctx 0x80329b400(www.cdwg.com/A)): created
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): start
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): try
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:14.707 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): query
16-May-2011 14:30:14.707 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): send
16-May-2011 14:30:14.707 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): sent
16-May-2011 14:30:14.707 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): senddone
16-May-2011 14:30:14.720 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): response
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): answer_response
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cache_message
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): clone_results
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): done
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): sendevents
16-May-2011 14:30:14.720 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:14.720 resolver: debug 3: fetch 0x80477dc40 (fctx 0x80329b400(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): shutdown
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:14.720 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): destroy
16-May-2011 14:30:15.746 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:15.746 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:15.746 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:15.746 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:15.746 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:15.747 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:15.747 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:15.772 resolver: debug 3: resquery 0x80340b2c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): response
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): answer_response
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): noanswer_response
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cache_message
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): clone_results
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): done
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): sendevents
16-May-2011 14:30:15.772 resolver: debug 3: fetch 0x80477d5c0 (fctx 0x803404800(www.cdwg.com.edgekey.net/A)): destroyfetch
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): shutdown
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): doshutdown
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:15.772 resolver: debug 3: fctx 0x803404800(www.cdwg.com.edgekey.net/A'): destroy
 
Code:
16-May-2011 14:30:17.334 queries: info: client 10.X.X.53#29058: query: www.cdwg.com IN A +
16-May-2011 14:30:17.334 security: debug 3: client 10.X.X.53#29058: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:17.334 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): create
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): join
16-May-2011 14:30:17.334 resolver: debug 3: fetch 0x801084800 (fctx 0x80329b400(www.cdwg.com/A)): created
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): start
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): try
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:17.334 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): query
16-May-2011 14:30:17.334 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): send
16-May-2011 14:30:17.334 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): sent
16-May-2011 14:30:17.334 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): senddone
16-May-2011 14:30:19.336 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): timeout
16-May-2011 14:30:19.336 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:19.336 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): try
16-May-2011 14:30:19.336 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): query
16-May-2011 14:30:19.336 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): send
16-May-2011 14:30:19.336 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): sent
16-May-2011 14:30:19.336 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): senddone
16-May-2011 14:30:19.385 resolver: debug 3: resquery 0x8032b2000 (fctx 0x80329b400(www.cdwg.com/A)): response
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): answer_response
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cache_message
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): clone_results
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): done
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): sendevents
16-May-2011 14:30:19.385 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): create
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): join
16-May-2011 14:30:19.385 resolver: debug 3: fetch 0x8010849e0 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): created
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): start
16-May-2011 14:30:19.385 resolver: debug 3: fetch 0x801084800 (fctx 0x80329b400(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): shutdown
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80329b400(www.cdwg.com/A'): destroy
16-May-2011 14:30:19.385 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:19.385 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:19.385 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:19.385 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:21.386 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): timeout
16-May-2011 14:30:21.386 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:21.386 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:21.386 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:21.386 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:21.386 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:21.386 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:21.396 resolver: debug 3: resquery 0x803429840 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): response
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): answer_response
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): noanswer_response
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cache_message
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): clone_results
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): done
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): sendevents
16-May-2011 14:30:21.396 resolver: debug 3: fetch 0x8010849e0 (fctx 0x80342d400(www.cdwg.com.edgekey.net/A)): destroyfetch
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): shutdown
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): doshutdown
16-May-2011 14:30:21.396 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:21.397 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:21.397 resolver: debug 3: fctx 0x80342d400(www.cdwg.com.edgekey.net/A'): destroy
 
Code:
16-May-2011 14:30:23.286 queries: info: client 10.X.X.53#21076: query: www.cdwg.com IN A +
16-May-2011 14:30:23.286 security: debug 3: client 10.X.X.53#21076: query (cache) 'www.cdwg.com/A/IN' approved
16-May-2011 14:30:23.286 resolver: debug 1: createfetch: www.cdwg.com A
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): create
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): join
16-May-2011 14:30:23.286 resolver: debug 3: fetch 0x801072ff0 (fctx 0x8032a9c00(www.cdwg.com/A)): created
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): start
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): try
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): getaddresses
16-May-2011 14:30:23.286 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): query
16-May-2011 14:30:23.286 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): send
16-May-2011 14:30:23.286 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): sent
16-May-2011 14:30:23.286 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): senddone
16-May-2011 14:30:25.287 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): timeout
16-May-2011 14:30:25.287 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:25.287 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): try
16-May-2011 14:30:25.287 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): query
16-May-2011 14:30:25.287 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): send
16-May-2011 14:30:25.287 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): sent
16-May-2011 14:30:25.287 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): senddone
16-May-2011 14:30:25.301 resolver: debug 3: resquery 0x803299580 (fctx 0x8032a9c00(www.cdwg.com/A)): response
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): answer_response
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): noanswer_response
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cache_message
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): clone_results
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cancelquery
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): done
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): sendevents
16-May-2011 14:30:25.301 resolver: debug 1: createfetch: www.cdwg.com.edgekey.net A
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): create
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): join
16-May-2011 14:30:25.301 resolver: debug 3: fetch 0x8010849f0 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): created
16-May-2011 14:30:25.301 resolver: debug 3: fetch 0x801072ff0 (fctx 0x8032a9c00(www.cdwg.com/A)): destroyfetch
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): shutdown
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): start
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): doshutdown
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): stopeverything
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): cancelqueries
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x8032a9c00(www.cdwg.com/A'): destroy
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): try
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): getaddresses
16-May-2011 14:30:25.301 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): query
16-May-2011 14:30:25.301 resolver: debug 3: resquery 0x803420840 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): send
16-May-2011 14:30:25.301 resolver: debug 3: resquery 0x803420840 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): sent
16-May-2011 14:30:25.301 resolver: debug 3: resquery 0x803420840 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): senddone
16-May-2011 14:30:25.343 resolver: debug 3: resquery 0x803420840 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): response
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): answer_response
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): noanswer_response
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): cache_message
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): clone_results
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): cancelquery
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): done
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): sendevents
16-May-2011 14:30:25.344 resolver: debug 3: fetch 0x8010849f0 (fctx 0x803404400(www.cdwg.com.edgekey.net/A)): destroyfetch
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): shutdown
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): doshutdown
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): stopeverything
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): cancelqueries
16-May-2011 14:30:25.344 resolver: debug 3: fctx 0x803404400(www.cdwg.com.edgekey.net/A'): destroy
...
16-May-2011 14:35:28.208 database: debug 1: decrement_reference: delete from rbt: 0x807248cc0 www.cdwg.com.edgekey.net
...
16-May-2011 14:35:48.306 database: debug 1: decrement_reference: delete from rbt: 0x80d643360 www.cdwg.com
...
 
TCPDUMP from Intranet_BIND toward Internet_Bind_01
Code:
INTRANET_BIND# tcpdump -vvv host 10.XXX.XXX.153
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
14:30:23.191310 IP (tos 0x0, ttl 64, id 59061, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].45245 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum ea61!] 39595+ A? www.cdwg.com. (30)
14:30:28.198979 IP (tos 0x0, ttl 64, id 59067, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].45245 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum ea61!] 39595+ A? www.cdwg.com. (30)
14:30:29.303071 IP (tos 0x0, ttl 62, id 42937, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]INTRANET_BIND[/b].45245: 39595 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:30.823271 IP (tos 0x0, ttl 64, id 59075, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].44228 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum e808!] 63398+ A? www.cdwg.com. (30)
14:30:35.827950 IP (tos 0x0, ttl 64, id 59077, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].44228 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum e808!] 63398+ A? www.cdwg.com. (30)
14:30:36.893379 IP (tos 0x0, ttl 62, id 44367, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]INTRANET_BIND[/b].44228: 63398 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:38.454977 IP (tos 0x0, ttl 64, id 59087, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].29058 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum 8754!] 59209+ A? www.cdwg.com. (30)
14:30:42.517939 IP (tos 0x0, ttl 62, id 45260, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]INTRANET_BIND[/b].29058: 59209 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:44.406689 IP (tos 0x0, ttl 64, id 59095, offset 0, flags [none], proto UDP (17), length 58)
    [b]INTRANET_BIND[/b].21076 > [b]INTERNET_BIND_01[/b].domain: [bad udp cksum 2399!] 49627+ A? www.cdwg.com. (30)
14:30:46.465254 IP (tos 0x0, ttl 62, id 46018, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]INTRANET_BIND[/b].21076: 49627 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]


TCPDUMP from INTERNET_BIND_01 toward Intranet_Bind
Code:
INTERNET_BIND_01# tcpdump -vvv host 10.X.X.53
tcpdump: listening on bce0, link-type EN10MB (Ethernet), capture size 96 bytes
14:30:02.070754 IP (tos 0x0, ttl 62, id 59061, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].45245 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 39595+ A? www.cdwg.com. (30)
14:30:07.078360 IP (tos 0x0, ttl 62, id 59067, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].45245 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 39595+ A? www.cdwg.com. (30)
14:30:08.182069 IP (tos 0x0, ttl 64, id 42937, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]10.X.X.53[/b].45245: 39595 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:09.246720 IP (tos 0x0, ttl 64, id 43182, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]10.X.X.53[/b].45245: 39595 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:09.702699 IP (tos 0x0, ttl 62, id 59075, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].44228 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 63398+ A? www.cdwg.com. (30)
14:30:14.707317 IP (tos 0x0, ttl 62, id 59077, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].44228 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 63398+ A? www.cdwg.com. (30)
14:30:15.772392 IP (tos 0x0, ttl 64, id 44367, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]10.X.X.53[/b].44228: 63398 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:17.334383 IP (tos 0x0, ttl 62, id 59087, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].29058 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 59209+ A? www.cdwg.com. (30)
14:30:21.396913 IP (tos 0x0, ttl 64, id 45260, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]10.X.X.53[/b].29058: 59209 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
14:30:23.286019 IP (tos 0x0, ttl 62, id 59095, offset 0, flags [none], proto UDP (17), length 58)
    [b]10.X.X.53[/b].21076 > [b]INTERNET_BIND_01[/b].domain: [udp sum ok] 49627+ A? www.cdwg.com. (30)
14:30:25.344237 IP (tos 0x0, ttl 64, id 46018, offset 0, flags [none], proto UDP (17), length 537)
    [b]INTERNET_BIND_01[/b].domain > [b]10.X.X.53[/b].21076: 49627 q: A? www.cdwg.com. 3/13/10 www.cdwg.com. [0s] CNAME[|domain]
 
DIGs results from INTRANETBIND using Internet_Bind_01 as forwarder/Resolver
Code:
[b]INTRANET_BIND[/b]# dig www.cdwg.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39595
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 10

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 20      IN      A       96.17.40.53

;; AUTHORITY SECTION:
.                       90755   IN      NS      a.root-servers.net.
.                       90755   IN      NS      c.root-servers.net.
.                       90755   IN      NS      e.root-servers.net.
.                       90755   IN      NS      b.root-servers.net.
.                       90755   IN      NS      l.root-servers.net.
.                       90755   IN      NS      g.root-servers.net.
.                       90755   IN      NS      k.root-servers.net.
.                       90755   IN      NS      d.root-servers.net.
.                       90755   IN      NS      i.root-servers.net.
.                       90755   IN      NS      m.root-servers.net.
.                       90755   IN      NS      f.root-servers.net.
.                       90755   IN      NS      h.root-servers.net.
.                       90755   IN      NS      j.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     196797  IN      A       198.41.0.4
a.root-servers.net.     350024  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     350101  IN      A       192.228.79.201
c.root-servers.net.     349990  IN      A       192.33.4.12
d.root-servers.net.     349968  IN      A       128.8.10.90
e.root-servers.net.     349968  IN      A       192.203.230.10
f.root-servers.net.     197779  IN      A       192.5.5.241
f.root-servers.net.     416395  IN      AAAA    2001:500:2f::f
g.root-servers.net.     349968  IN      A       192.112.36.4
h.root-servers.net.     364239  IN      A       128.63.2.53

;; Query time: 1104 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May 16 14:30:29 2011
;; MSG SIZE  rcvd: 509

[b]INTRANET_BIND[/b]# dig www.cdwg.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63398
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 10

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 13      IN      A       96.17.40.53

;; AUTHORITY SECTION:
.                       90748   IN      NS      e.root-servers.net.
.                       90748   IN      NS      a.root-servers.net.
.                       90748   IN      NS      h.root-servers.net.
.                       90748   IN      NS      b.root-servers.net.
.                       90748   IN      NS      k.root-servers.net.
.                       90748   IN      NS      d.root-servers.net.
.                       90748   IN      NS      m.root-servers.net.
.                       90748   IN      NS      c.root-servers.net.
.                       90748   IN      NS      g.root-servers.net.
.                       90748   IN      NS      j.root-servers.net.
.                       90748   IN      NS      i.root-servers.net.
.                       90748   IN      NS      l.root-servers.net.
.                       90748   IN      NS      f.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     196790  IN      A       198.41.0.4
a.root-servers.net.     350017  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     350094  IN      A       192.228.79.201
c.root-servers.net.     349983  IN      A       192.33.4.12
d.root-servers.net.     349961  IN      A       128.8.10.90
e.root-servers.net.     349961  IN      A       192.203.230.10
f.root-servers.net.     197772  IN      A       192.5.5.241
f.root-servers.net.     416388  IN      AAAA    2001:500:2f::f
g.root-servers.net.     349961  IN      A       192.112.36.4
h.root-servers.net.     364232  IN      A       128.63.2.53

;; Query time: 1065 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May 16 14:30:36 2011
;; MSG SIZE  rcvd: 509
 
Code:
[b]INTRANET_BIND[/b]# dig www.cdwg.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59209
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 10

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 7       IN      A       96.17.40.53

;; AUTHORITY SECTION:
.                       90742   IN      NS      f.root-servers.net.
.                       90742   IN      NS      d.root-servers.net.
.                       90742   IN      NS      m.root-servers.net.
.                       90742   IN      NS      i.root-servers.net.
.                       90742   IN      NS      l.root-servers.net.
.                       90742   IN      NS      a.root-servers.net.
.                       90742   IN      NS      c.root-servers.net.
.                       90742   IN      NS      g.root-servers.net.
.                       90742   IN      NS      b.root-servers.net.
.                       90742   IN      NS      h.root-servers.net.
.                       90742   IN      NS      j.root-servers.net.
.                       90742   IN      NS      k.root-servers.net.
.                       90742   IN      NS      e.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     196784  IN      A       198.41.0.4
a.root-servers.net.     350011  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     350088  IN      A       192.228.79.201
c.root-servers.net.     349977  IN      A       192.33.4.12
d.root-servers.net.     349955  IN      A       128.8.10.90
e.root-servers.net.     349955  IN      A       192.203.230.10
f.root-servers.net.     197766  IN      A       192.5.5.241
f.root-servers.net.     416382  IN      AAAA    2001:500:2f::f
g.root-servers.net.     349955  IN      A       192.112.36.4
h.root-servers.net.     364226  IN      A       128.63.2.53

;; Query time: 4063 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May 16 14:30:42 2011
;; MSG SIZE  rcvd: 509

[b]INTRANET_BIND[/b]# dig www.cdwg.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @10.XXX.XXX.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49627
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 10

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 3       IN      A       96.17.40.53

;; AUTHORITY SECTION:
.                       90738   IN      NS      m.root-servers.net.
.                       90738   IN      NS      l.root-servers.net.
.                       90738   IN      NS      j.root-servers.net.
.                       90738   IN      NS      d.root-servers.net.
.                       90738   IN      NS      a.root-servers.net.
.                       90738   IN      NS      i.root-servers.net.
.                       90738   IN      NS      e.root-servers.net.
.                       90738   IN      NS      c.root-servers.net.
.                       90738   IN      NS      h.root-servers.net.
.                       90738   IN      NS      b.root-servers.net.
.                       90738   IN      NS      f.root-servers.net.
.                       90738   IN      NS      k.root-servers.net.
.                       90738   IN      NS      g.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     196780  IN      A       198.41.0.4
a.root-servers.net.     350007  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     350084  IN      A       192.228.79.201
c.root-servers.net.     349973  IN      A       192.33.4.12
d.root-servers.net.     349951  IN      A       128.8.10.90
e.root-servers.net.     349951  IN      A       192.203.230.10
f.root-servers.net.     197762  IN      A       192.5.5.241
f.root-servers.net.     416378  IN      AAAA    2001:500:2f::f
g.root-servers.net.     349951  IN      A       192.112.36.4
h.root-servers.net.     364222  IN      A       128.63.2.53

;; Query time: 2058 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May 16 14:30:46 2011
;; MSG SIZE  rcvd: 509
 
DIGs results from Internet_Bind_01 using itself as forwarder/Resolver
Code:
INTERNET_BIND_01# dig www.cdwg.com @10.XXX.XXX.153

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @10.3.2.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48189
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 10

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 20      IN      A       96.16.248.53

;; AUTHORITY SECTION:
.                       87461   IN      NS      i.root-servers.net.
.                       87461   IN      NS      l.root-servers.net.
.                       87461   IN      NS      m.root-servers.net.
.                       87461   IN      NS      f.root-servers.net.
.                       87461   IN      NS      c.root-servers.net.
.                       87461   IN      NS      b.root-servers.net.
.                       87461   IN      NS      h.root-servers.net.
.                       87461   IN      NS      k.root-servers.net.
.                       87461   IN      NS      e.root-servers.net.
.                       87461   IN      NS      a.root-servers.net.
.                       87461   IN      NS      g.root-servers.net.
.                       87461   IN      NS      j.root-servers.net.
.                       87461   IN      NS      d.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     193503  IN      A       198.41.0.4
a.root-servers.net.     346730  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     346807  IN      A       192.228.79.201
c.root-servers.net.     346696  IN      A       192.33.4.12
d.root-servers.net.     346674  IN      A       128.8.10.90
e.root-servers.net.     346674  IN      A       192.203.230.10
f.root-servers.net.     194485  IN      A       192.5.5.241
f.root-servers.net.     413101  IN      AAAA    2001:500:2f::f
g.root-servers.net.     346674  IN      A       192.112.36.4
h.root-servers.net.     360945  IN      A       128.63.2.53

;; Query time: 4258 msec
;; SERVER: 10.XXX.XXX.153#53(10.XXX.XXX.153)
;; WHEN: Mon May 16 15:25:02 2011
;; MSG SIZE  rcvd: 509


DIGs results from Internet_Bind_01 using ISP DNS Servers as forwarder/Resolver
Code:
INTERNET_BIND_01# dig www.cdwg.com @68.87.68.162

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @68.87.68.162
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32509
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 20      IN      A       184.86.56.53

;; Query time: 17 msec
;; SERVER: 68.87.68.162#53(68.87.68.162)
;; WHEN: Mon May 16 15:25:30 2011
;; MSG SIZE  rcvd: 117

INTERNET_BIND_01# dig www.cdwg.com @68.87.74.162

; <<>> DiG 9.6.-ESV-R3 <<>> www.cdwg.com @68.87.74.162
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28437
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cdwg.com.                  IN      A

;; ANSWER SECTION:
www.cdwg.com.           0       IN      CNAME   www.cdwg.com.edgekey.net.
www.cdwg.com.edgekey.net. 0     IN      CNAME   e1973.b.akamaiedge.net.
e1973.b.akamaiedge.net. 20      IN      A       96.16.248.53

;; Query time: 39 msec
;; SERVER: 68.87.74.162#53(68.87.74.162)
;; WHEN: Mon May 16 15:25:36 2011
;; MSG SIZE  rcvd: 117


Stats from the two servers in question:
Code:
[b]INTERNET_BIND_01[/b]# rndc status
version: 9.6.-ESV-R3 (BUGGER OFF)
CPUs found: 16
worker threads: 16
number of zones: 79
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 175/0/1000
tcp clients: 0/100
server is up and running
[b]INTERNET_BIND_01[/b]# vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr mf0 mf1   in   sy   cs us sy id
 1 0 0    650M    10G     5   0   0   0    79   9   0   0  306 3017  441  0  0 100
Code:
[b]INTRANET_BIND[/b]# rndc status
version: 9.6.-ESV-R3 (BUGGER OFF)
CPUs found: 1
worker threads: 1
number of zones: 35
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running
[b]INTRANET_BIND[/b]# vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy id
 2 0 0    435M  7700M     4   0   0   0     4   0   0   0    3   33  236  0  0 100
 
I do a dig from my internal BIND DNS server for an Internet FQDN

Hi quick question, When you mention from the internal Bind DNS server, this is from within the Intranet right?

And your /etc/resolv.conf has:

Code:
127.0.0.1
Internet DNS #1
Internet DNS #2
 
The Intranet Bind server's /etc/resolv.conf reads as follows:
Code:
# Name Server
nameserver       10.x.x.53
# nameserver     10.XXX.XXX.153
# nameserver     10.XXX.XXX.154

It's only pointed at itself as its BIND service has forwarders which dictate where traffic is allowed to resolve. The Internet Bind Servers were used previously for testing and were commented out afterward. Are you aiming at suggesting that the server is attempting to go outside of itself and then back in for resolution due to the fact that I am using its network IP instead of the 127 IP?

Internet Bind only has the ISP's DNS servers for resolution. It does not have itself listed as a nameserver. Again, should it list itself (127 IP) along with the ISP DNS servers?

Could the issue and extreme latency be due to that misconfig and I be that n00b-ish?
 
w3nd377 said:
The Intranet Bind server's /etc/resolv.conf reads as follows:
Code:
# Name Server
nameserver       10.x.x.53
# nameserver     10.XXX.XXX.153
# nameserver     10.XXX.XXX.154

Its only pointed at itself as its BIND service has forwarders which dictate where traffic is allowed to resolve. The Internet Bind Servers were used previously for testing and were commented out afterward. Are you aiming at suggesting that the server is attempting to go outside of itself and then back in for resolution due to the fact that I am using its network IP instead of the 127 IP?

Internet Bind only has the ISP's DNS servers for resolution. It does not have itself listed as a nameserver. Again, should it list itself (127 IP) along with the ISP DNS servers?

Could the issue and extreme latency be due to that misconfig and I be that n00b-ish?

Hi, preliminarily I was thinking it might have something to do with the root-hints configuration on the intranet resolver: are custom roots being used instead? If not, is it possible the extra latency might be due to that?

As far as listing 127 first, the example named.conf that is installed on my FreeBSD laptop states:

Code:
// If you enable a local name server, don't forget to enter 127.0.0.1
// first in your /etc/resolv.conf so this server will be queried.

I'm by no means an expert and I have to confess I didn't analyze your tcpdump, but the root-hints config and maybe the 127 issue might be troubleshooting steps you might verify at least to get out of the way. Also, if it isn't either of these issues, it still might be easier to troubleshoot your config if you posted an anonymized named.conf for instance. Regards
 
Root hints are only enabled on the Internet_BIND_01/02. Root hints are not enabled on the Intranet_BIND. Intranet_BIND is only a forwarder/cache...same with Internet_BIND_01/02.

When I dig from the Internet_BIND_01 or 02 toward the ISPs DNS servers the response is approximately +/-40msec. However when I do a dig against itself to use its BIND service which has global forwarding to the ISP's DNS servers, the response back is random at best....35msec, +2000msec, +4000msec, or times out....even if I do the dig repeatedly one after another when (and if) results are returned.

I would assume BIND should not be having issues retrieving data out of cache if the server has 48GB RAM and four quadcore CPUs to use just for that service? Bluecats biggest box only has ~4GB ram and a single quad-core CPU and they run BIND inside on Debian Linux.

Here is a summary of the config options:
Code:
options {
// All file and path names are relative to the chroot directory,
// if any, and should be fully qualified.
directory       "/etc/namedb/working";
pid-file        "/var/run/named/pid";
dump-file       "/var/dump/named_dump.db";
statistics-file "/var/stats/named.stats.txt";
recursion yes;
// Restricting the systems which can query this server
allow-query { "localhost"; "ACL1"; "ACL2"; "ACL3"; "ACL4"; "ACL5"; };
allow-query-on {10.XXX.XXX.153; };
interface-interval 0;
version "BUGGER OFF";
forward only;
forwarders { 68.87.68.162; 68.87.74.162; ];
query-source address * port 53;
];
Should a limit be placed on the cache? Default is unlimited. Does there need to be a specific option regarding the cache or its size? Reason I ask is that the cdwg.com records from earlier post were removed from the rbt (red-black-tree) database after 5 minutes....unsure what its TTL was through.
Code:
(FROM POST AT 15:56)
...
16-May-2011 14:35:28.208 database: debug 1: decrement_reference: delete from rbt: 0x807248cc0 www.cdwg.com.edgekey.net
...
16-May-2011 14:35:48.306 database: debug 1: decrement_reference: delete from rbt: 0x80d643360 www.cdwg.com
...
 
Resolution found

DANGER WILL ROBINSON!!!!

I worked with my manager and we went over the config with a fine tooth comb and found a misconfig. O'Reily books fail yet again. We commented out the line regarding query-source and all is happy and the Internet is now faster then before. The query-source statement option as it was configured was doing a DOS against named resolution. It was forcing all traffic to go over port 53 however when the port was already in use the server and/or the firewall was forcing following traffic to wait its turn. Thus a DOS on name resolution and the random latency of digs and nslookups.

Line removed:
Code:
query-source address * port 53;

Lessons Learned:
1) O'Reily as good as they may be, are NOT bibles or techie religious material to always believe.
2) query-source address * port 53 could be used but must be used with extreme caution and understanding that it could DOS you or your organization.
3) [ unrelated ] BIND puts out better logs then Bill's DNS
 
Hi w3nd377,

Thanks for the heads up. The sample config file that comes with FreeBSD states that option should only be used as a last resort to get around a very restrictive firewall, and that modern versions of BIND use a random UDP port for each outgoing query by default. Never used the query-source address option myself, so this is good to know!

Glad things are working out for you now. Regards
 
Back
Top