NAT on local traffic which does not flow through the interface?

[rf10]

I have a "host A" with two network interfaces, em0 (externally-facing), 10.0.0.1/8 and em1 (lan, or internal side), 192.168.1.1/24. I have two apps "app.example.org:8080" and "web.example.org:80" which are run on two separate boxes on the 192.168.1.1/24 subnet. To expose them externally, external DNS is configured to resolve "app.example.org" and "web.example.org" to 10.0.0.1, and since they run on different ports, I use NAT on em0 and port-forward the requests to the correct box on the lan side. This is a typical setup, nothing unusual about it, and it works as expected.

There are two annoying problems with this setup:

1. When "host A" tries to connect "app.example.org", it resolves to 192.168.1.1, so it does not go through the firewall's NAT translation/port forward.
2. Similarly, when host "app.example.org" tries to connect to "web.example.org", it resolves to 192.168.1.1, and we go to case (1).

The fix is to provide alternative DNS resolution so that the names are resolved to the appropriate lan side address. I confirmed that this works by hard coding entries in the hosts file. Obviously, to scale this, an "internal" DNS server needs to be configured, which seems like a considerable overhead for the otherwise simple(?!) networking solution.

I played a bit with the firewall configuration, but neither PF nor IPFW worked. For the traffic which originates on "host A" it does not actually travel through the em0 interface, and thus NAT and port forwarding rules don't get applied. There is verbiage in the PF man pages which explicitly states that NAT wouldn't get applied to the traffic which does not pass through the interface; I didn't see any similar warnings in IPFW, but my practical experiments confirmed that IPFW appear to behave the same way.

Do you have any ideas which rely on a networking solution to this problem?

 

[SirDice]

Run it through a reverse proxy like net/haproxy. Then both can run on port 80, HAProxy can switch backends based on the HTTP Host header. Added bonus, you don't need NAT or redirections. And the applications can be accessed internally on their 'external' address since the traffic is proxied.

 

[rf10]

Thanks. I am trying to rationalize this how it is better than the internal DNS solution. No NAT is nice, though.

 

[rf10]

So, I just wanted to say thanks again. I had a chance to clear my head and think this through, and the proxy solution is a winner for this use case, for a number of reasons:
1. No NAT
2. Simple DNS setup (no change, actually)
3. Automation of haproxy config and hot reloading is probably simpler than the alternative DNS (or even firewall solution). It probably does require a custom "side-car" software component which would consult DNS/or directory to locate the services/endpoints to expose, rewrite haproxy config, and reload it. I remember seeing something like this already implemented.