Please share this information with developers to raise awareness of this core infrastructure issue, and stabilize FQDN handling in firewalls for today's cloud-based internet.
Cloud load balancing broke this model without changing the spec. Modern providers answer each query with a different set of A-records for the same FQDN. api.example.com might resolve to [10.0.0.1, 10.0.0.2] on one query and [10.0.0.3, 10.0.0.4] on the next, all within seconds.
The consequence: When a traditional DNS cache receives the second response, it doesn't add the new IPs to the list. It replaces the old list. The cache follows the specification's original intent: each new authoritative answer supersedes the previous one. The old IPs disappear from the cache immediately, even though clients in flight might still be connecting to them.
Now apply this to a firewall:
However, the critical question is: Does the problem still exist in modern firewalls? Vendors may have implemented workarounds, changed their caching semantics, or addressed this issue without public documentation. Re-testing against current platforms (Palo Alto Networks, Fortinet, OPNsense, etc.) is essential.
Rather than conducting vendor-sponsored research myself, I'm inviting the FreeBSD community to participate in collaborative validation:
This was in production for 4 years, serving 13 users with a whitelist of nearly 250 primary domains and over 1,000 subdomains, including Office 365, Google's full suite of services and Salesforce. The solution works, but this is not a sales pitch. If the pf version were complete, it would be a free release right now.
What questions do you have?
Resolving DNS-Cache Replacement vs. Load-Balanced Concurrency
Abstract
Traditional DNS caches—whether on clients, firewalls, or servers—implement the DNS specification's original semantics: each new response replaces the previous one rather than accumulating with it. However, cloud providers use DNS load balancing by returning different sets of A-records on each query, with very short TTLs. In the firewall's DNS cache, this causes a fundamental synchronization failure: when a rule references an FQDN, the IPs it permits are constantly out of step with the IPs clients are actually connecting to. The problem is not retention—it's mutually exclusive cache replacement in a model that requires concurrent validity. I present a DNS caching layer that enforces concurrent A-record aggregation and synchronizes the complete set with layer-3 firewall rules in near-real-time. Originally validated on SonicWall TZ-series firewalls in 2017, "Gatekeeper" is now being redesigned for FreeBSD's pf firewall using direct kernel struct manipulation. This post presents the architecture, seeks validation that the problem persists in modern firewalls, and invites the community to use provided testing tools to collaboratively gather that evidence.The Problem: DNS Semantics vs. Load-Balancing Reality
The DNS specification defines how to cache a single, authoritative answer: when a record's TTL expires, remove it from cache. Simple, clean, designed for a world where one query = one answer.Cloud load balancing broke this model without changing the spec. Modern providers answer each query with a different set of A-records for the same FQDN. api.example.com might resolve to [10.0.0.1, 10.0.0.2] on one query and [10.0.0.3, 10.0.0.4] on the next, all within seconds.
The consequence: When a traditional DNS cache receives the second response, it doesn't add the new IPs to the list. It replaces the old list. The cache follows the specification's original intent: each new authoritative answer supersedes the previous one. The old IPs disappear from the cache immediately, even though clients in flight might still be connecting to them.
Now apply this to a firewall:
- Firewall queries api.example.com and caches [10.0.0.1, 10.0.0.2]
- Seconds later, a client tries to connect to 10.0.0.3
- Firewall blocks it (not in cache)
- Meanwhile, the firewall's DNS cache has already been replaced with [10.0.0.3, 10.0.0.4]
- The rule for api.example.com now permits IPs the client may never use, and blocks IPs the client is actively using
Why This Breaks Everything, Not Just Deny-by-Default
The symptoms vary by firewall strategy, but the synchronization failure affects all of them:- Deny-by-Default (Whitelist): Clients get "connection refused" because their legitimate IPs are no longer in the cache.
- Blacklisting/Deny Rules: You're blocking obsolete IPs while allowing newly rotated ones (which may be owned by different tenants or attackers).
- Logging and Attribution: Your firewall logs show traffic to IP X, but your FQDN rules reference a different cached set. Incident investigation becomes guesswork.
- Content Filtering and DLP: You enforce a rule on "facebook.com," but you're inspecting (or missing) the wrong traffic due to stale cached IPs.
- Threat Intelligence and IDS: Rules designed to block malicious domains become unreliable when A-records rotate faster than cache validity.
The Solution: Concurrent Aggregation and Synchronization
Gatekeeper solved this by rejecting the cache-replacement model entirely. Instead of replacing, it aggregates:- Enforce Concurrent Validity: For each FQDN, Gatekeeper maintains a rolling 24-hour window of all unique A-records ever returned by upstream resolvers. If 10.0.0.1 appears once and 10.0.0.2 appears later, both remain valid simultaneously until their 24-hour window closes.
- Synchronize with the Firewall: Whenever new A-records are observed, Gatekeeper pushes the complete aggregated set to the firewall's layer-3 rules for that FQDN. The firewall now permits the union of all IPs clients might see.
- Manage State: A state machine holds DNS responses to clients until the firewall has been updated with the corresponding A-records, eliminating the window where clients and firewalls diverge.
Architecture
Original Implementation (2017)
Gatekeeper was built for SonicWall TZ-series firewalls. It used SSH to push aggregated IP sets into firewall rules—functional but slow (firewall responses took 2–5 seconds). This version validated the core concept: the DNS/firewall synchronization problem is real, fixable, and worth solving.New Implementation: FreeBSD [pf]
The redesign targets FreeBSD's pf firewall because:- Speed: Instead of SSH and CLI commands, Gatekeeper updates rules by directly manipulating C structs in kernel memory, achieving synchronization in milliseconds.
- Native Design: pf already supports dynamic table updates; Gatekeeper leverages this native capability.
- Community Alignment: FreeBSD values correctness, clarity, and solving real problems elegantly.
Status and Community Validation
The pf implementation is pre-beta. The architecture is proven (validated on SonicWall in 2017), and the core design is complete, but production testing is incomplete, and active development is paused.However, the critical question is: Does the problem still exist in modern firewalls? Vendors may have implemented workarounds, changed their caching semantics, or addressed this issue without public documentation. Re-testing against current platforms (Palo Alto Networks, Fortinet, OPNsense, etc.) is essential.
Rather than conducting vendor-sponsored research myself, I'm inviting the FreeBSD community to participate in collaborative validation:
- I will provide testing tools that community members can run on their environments to probe whether their firewalls exhibit cache-replacement behavior.
- These tools will generate results in a standardized format, allowing us to gather evidence across diverse firewall models and configurations.
- Shared findings will inform whether Gatekeeper is addressing a persistent problem or a historical one, and will guide the final implementation.
This was in production for 4 years, serving 13 users with a whitelist of nearly 250 primary domains and over 1,000 subdomains, including Office 365, Google's full suite of services and Salesforce. The solution works, but this is not a sales pitch. If the pf version were complete, it would be a free release right now.
What questions do you have?