IPFW rules for DNS server: Unbound or Bind

I run a recursive DNS server in a jail on a FreeBSD 11 box, the host system itself is running an IPFW firewall that allows all outbound connections but filters inbound based on the below rules.

The DNS server will accept queries from the jail host or other jails on the box, but it won't accept connections from other machines on my LAN.

The server config is not the problem. I've tried Unbound and BIND9.11 servers allowing for queries from the LAN and localhost. Both servers will fail to accept connections from other systems on the LAN, they time out with the below error.

; <<>> DiG 9.8.3-P1 <<>> @
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
What's leading me to thinking this is an issue with my IPFW firewall is if I setup Unbound or Bind on a virtual machine running with no firewall, it will accept connections with exactly the same config file for Unbound or Bind when it wouldn't accept connections when sitting behind IPFW.

This isn't specific to my LAN setup either, this exact issue is also occurring if I setup Unbound on an Internet-facing virtual server with IPFW running. This occurs if Bind or Unbound is running in a jail or not.

Am I missing something from my IPFW rules? Rules below:

cmd="ipfw add"

# Ban tables
$cmd 00030 deny ip from "table(jailed)" to me
$cmd 00031 deny ip from "table(jailed-perm)" to me

# Statefull firewall config, more secure
$cmd 00060 check-state

# Allow outbound traffic
$cmd 00130 allow ip from me to any keep-state

$cmd 11020 allow tcp from any to me dst-port 22 setup keep-state

$cmd 11027 allow udp from any to me dst-port 53 setup keep-state
$cmd 11028 allow tcp from any to me dst-port 53 setup keep-state

# General Network - ICMP & IGMP
$cmd 61001 allow icmp from any to any
$cmd 61002 allow igmp from any to any

# Deny the rest
$cmd 65500 deny ip from any to any
There's not much to it than allowing access to port 53 (both UDP and TCP) for DNS. But it's fairly simple to rule out the firewall, just switch it off and see if you still get the time-outs.
I don't know why I didn't try it with the firewall off before. Turning IPFW off fixes the issue.

I can't think of any reason why IPFW is not allowing this traffic through seeing as all outbound connections are allowed, and TCP & UDP on 53 are let through also.

My rc.conf has firewall_quiet="YES" in it, I'm not sure if that makes a difference.

Are there any other dependencies that I'm not taking into account with my IPFW rules?