nameserver for * intermitently doesn't respond

After coming back to one of, or my nameserver reports remote server failure.

I did some packet dumps on my nameserver and from what I can tell so far this is what happens:

1. DNS request goes out over UDP to
2. responds with truncated flag
3. DNS request goes out over TCP to
4. TCP connection times out

The odd thing is that after a couple of refreshes in the browser it start working. Doing a packet dump after clearing my nameserver cache shows that responds via UDP no problem.

I've also tried using dig against using tcp and udp and IPv4 and IPv6 and everything works.

I'm still trying to get more packet captures to see if it's isolated to IPv4 or IPv6 but it's not easy because it's intermittent.

Any ideas?