Inter-jail communication - routing problem, traffic backwards?

[perkypork]

I am having a bit of trouble working out how to solve the issue that I am having and I was hoping someone could point me in the correct direction.

1. I am using multiple jails behind a HTTP proxy to accept traffic and send it to the correct jail.
2. I am using IPFW kernel NAT which is working fine - (https://forums.freebsd.org/threads/nat-and-ipfw-cannot-get-it-to-work.53553/#post-302532)
3. The proxy and the jails share the same public IP.
4. All jails have an IP on the 10.0.0.0/16 subnet using a cloned loopback (lo1)

Everything is working fine except for one issue:
The proxy (nginx) receives traffic for jail1.example.com but jail1.example.com is configured to proxy_pass the traffic to jail2.example.com for this particular url. Nginx (on jail1.example.com) resolves the IP of jail2.example.com (which is the public IP of the proxy and jails) and sends the traffic over to jail2.example.com correctly. The traffic leaves jail1.example.com for jail2.example.com and is intercepted by FreeBSD. This is where is goes bad, FreeBSD sees the traffic as a response to the initial incoming traffic to jail1.example.com instead of new traffic from jail1.example.com to jail2.example.com, so it sends the traffic to jail2.example.com on the same high port it come into jail1.example.com. jail2.example.com can only accept traffic on port 443, so IPFW blocks the traffic as its destination is some high port. I am hoping you understood the explanation, if not I can post some IPFW log traffic to show what is happening.

If you miraculously understood my explanation, do you happen to know how I might fix this issue so that things work correctly?

 

[neogeo]

Could there be an issue about network address translation? In PF at least, there's a keyword keep-state, available to some of the PF packet filtering commands.

Considering IPFW, the manual page ipfw(8) features a section, titled "Network Address Translation (NAT)". Without beginning a study about ipfw configuration syntax and packet filtering, in my comment, perhaps the manual page may help to provide any more of an introspective view of the configuration?

 

[perkypork]

Could there be an issue about network address translation? In PF at least, there's a keyword keep-state, available to some of the PF packet filtering commands.

Considering IPFW, the manual page ipfw(8) features a section, titled "Network Address Translation (NAT)". Without beginning a study about ipfw configuration syntax and packet filtering, in my comment, perhaps the manual page may help to provide any more of an introspective view of the configuration?

I am almost certain its not NAT as I can see that the traffic doesn't hit NAT, it hits the network stack (if that is the right terminology) and is treated as a response.

 

[perkypork]

I will have another test box up and running today and will capture some IPFW logs and post them. Hoping it might shed a little more light on my issue and someone might be able to point me in the correct direction.

 

[perkypork]

Below is the traffic for a single request, which fails because the traffic is being mucked up.

The Proxy jail is 10.0.0.1, which intercepts all traffic hitting the host. The Proxy jail (based on URL) forwards the traffic to a specific HTTP app Jail. In this particular case the HTTP app jail is 10.0.0.3. The 10.0.0.3 jail is configured to do a curl request to another HTTP app jail (10.0.0.25) and then return the results to the proxy and then my laptop. I have added comments to the logs (every rule in IPFW is logged).

IP aaa.bbb.ccc.ddd is the originating IP (in this case my laptop which initiates the request).

IP www.xxx.yyy.zzz is the NAT IP, which all jails including the proxy sit behind.

Third HTTP app jail IP is 10.0.0.25. This is the jail the traffic should be hitting but it isn't. The traffic should be leaving 10.0.0.3 bound for www.xxx.yyy.zzz:443, which hits the proxy and is then forwarded to 10.0.0.25. This is hairpin or loopback NAT, which does not seem to be working in this case.

# Command to get the output below
root@host5:/ # tail -f /var/log/security | grep -Ew "10\.0\.0\.1|10\.0\.0\.3"

# Traffic from my laptop hits the Host and NAT forwards the traffic to the Proxy jail.
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:03 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0

# Proxy jail forwards the traffic to the HTTP app jail
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1

# HTTP app jail communcates with its installation of PHP-FPM on port 9000
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.3:28618 8.8.8.8:53 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 123 SkipTo 65510 UDP 10.0.0.3:28618 8.8.8.8:53 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65510 Nat UDP 10.0.0.3:28618 8.8.8.8:53 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1

# Proxy and laptop do some communicating
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0

# HTTP app jail does DNS lookup
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.3:28618 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 123 SkipTo 65510 UDP 8.8.8.8:53 10.0.0.3:28618 in via bce0
Nov 17 07:31:04 host5 kernel: ipfw: 65520 Accept UDP 8.8.8.8:53 10.0.0.3:28618 in via bce0

# This is where things go wrong. The traffic is backwards, it should show 10.0.0.3:16936 -> www.xxx.yyy.zzz:443. Not sure why the traffic is showing as backwards.
Nov 17 07:31:04 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:04 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:07 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:07 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:10 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:10 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:14 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:14 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:17 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:17 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:20 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:20 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:26 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:26 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:39 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:31:39 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:32:03 host5 kernel: ipfw: 101 UNKNOWN TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1
Nov 17 07:32:03 host5 kernel: ipfw: 65501 Deny TCP www.xxx.yyy.zzz:443 10.0.0.3:16936 out via lo1

# HTTP app jail and proxy communicate. HTTP app jail tells proxy it failed.
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1

# Proxy communicates with my laptop again
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0

# HTTP app jail does some cleanup communication with its PHP-fpm install.
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:42292 10.0.0.3:9000 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 208 Accept TCP 10.0.0.3:9000 10.0.0.3:42292 in via lo1

# Proxy and App Jail talk again
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.3:443 10.0.0.1:45833 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 out via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1
Nov 17 07:32:04 host5 kernel: ipfw: 202 Accept TCP 10.0.0.1:45833 10.0.0.3:443 in via lo1

# Proxy sends a 504 to my laptop.
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65510 Nat TCP 10.0.0.1:443 aaa.bbb.ccc.ddd:35810 out via bce0

# Laptop responds to Proxy jail
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 101 UNKNOWN TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 120 SkipTo 65510 TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0
Nov 17 07:32:04 host5 kernel: ipfw: 65520 Accept TCP aaa.bbb.ccc.ddd:35810 10.0.0.1:443 in via bce0

 

[perkypork]

I am pretty certain that I can solve this issue by having split DNS but I was hoping for this not to be required as it creates other complex requirements on the system. I would ideally like each HTTP app jail to operate like it was a standalone server. If it needs to communicate with another Jail the traffic traverses NAT and then comes back in again to the other Jail through the proxy. I think it might be something to do with my routing (which I have no modified manually at all), i.e. I might need to do some manual route manipulation but I am not sure what I should add to get it to work (if it will work at all).

 

[perkypork]

This is the output of netstat -rn

xxx.xxx.xxx.xxx is the gateway IP
aaa.aaa.aaa.aaa is the broadcast IP
www.xxx.yyy.zzz is the NAT IP (all traffic to this address on port 443 is forwarded to proxy)
yyy.yyy.yyy.yyy is the Host IP (the host traffic is separate to all jail and NAT traffic)

root@host5:/ # netstat -rn
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
default  xxx.xxx.xxx.xxx  UGS  bce0
10.0.0.1  link#6  UH  lo1
10.0.0.2  link#6  UH  lo1
10.0.0.3  link#6  UH  lo1
10.0.0.4  link#6  UH  lo1
10.0.0.5  link#6  UH  lo1
10.0.0.6  link#6  UH  lo1
10.0.0.7  link#6  UH  lo1
10.0.0.8  link#6  UH  lo1
10.0.0.9  link#6  UH  lo1
10.0.0.10  link#6  UH  lo1
10.0.0.11  link#6  UH  lo1
10.0.0.12  link#6  UH  lo1
10.0.0.13  link#6  UH  lo1
10.0.0.14  link#6  UH  lo1
10.0.0.15  link#6  UH  lo1
10.0.0.16  link#6  UH  lo1
10.0.0.17  link#6  UH  lo1
10.0.0.18  link#6  UH  lo1
10.0.0.19  link#6  UH  lo1
10.0.0.20  link#6  UH  lo1
10.0.0.21  link#6  UH  lo1
10.0.0.22  link#6  UH  lo1
10.0.0.23  link#6  UH  lo1
10.0.0.24  link#6  UH  lo1
10.0.0.25  link#6  UH  lo1
10.0.0.26  link#6  UH  lo1
aaa.aaa.aaa.aaa/25  link#1  U  bce0
www.xxx.yyy.zzz  link#1  UHS  lo0
www.xxx.yyy.zzz/32  link#1  U  bce0
yyy.yyy.yyy.yyy  link#1  UHS  lo0
127.0.0.1  link#5  UH  lo0

Internet6:
Destination  Gateway  Flags  Netif Expire
::/96  ::1  UGRS  lo0
::1  link#5  UH  lo0
::ffff:0.0.0.0/96  ::1  UGRS  lo0
fe80::/10  ::1  UGRS  lo0
fe80::%lo0/64  link#5  U  lo0
fe80::1%lo0  link#5  UHS  lo0
ff01::%lo0/32  ::1  U  lo0
ff02::/16  ::1  UGRS  lo0
ff02::%lo0/32  ::1  U  lo0

 

[perkypork]

I am pretty certain that I can solve this issue by having split DNS

I was wrong about this. I am using nginx as a layer 7 proxy as different web apps sit behind different folders (example.com/app/appnumber1)