Solved Internally connecting to jail uses jail's IP as both source and target

I'm still quite new to FreeBSD, so I hope I didn't make some stupid mistake or miss something obvious.

I'm running multiple jails on my VPS. They all use the interface lo1 that has multiple IPs assigned to it: 192.168.2.1 for the host and 192.168.2.11, 192.168.2.12 etc. for the jails.
No jail binds to 192.168.2.1, that is the internal address that I want to use for the host.

The jails can connect to the outside world (PF handles NAT) and connecting to them from outside works too.

All connections to the jails go through the host, so, to the jails, they should look like they're coming from the host's internal address.
However, both the source and target IP will always be the jail's IP: When connecting to a jail that uses the IP 192.168.2.11, the connection will be originating from 192.168.2.11 too. How can I set this up so connections originate from another internal address that just the host uses, e.g. 192.168.2.1?

I can solve this for nginx (which I'm using to proxy connections to the jails) using the proxy_bind directive, but that seems like a cheap workaround to me, considering it doesn't cover other services. I feel like there has to be a system-level solution.
 
Very similar setup here. However, I run haproxy to load balance (which has a nice property of being able to TLS termination for multiple nginx instances running in different jails). In this setup, I am pretty sure that nginx boxes see the traffic coming from the haproxy IP, which is, if I read correctly, you are trying to accomplish. I think there is special configuration in haproxy to preserve the original requestor's ip address, but I couldn't be bothered.
 
I am pretty sure that nginx boxes see the traffic coming from the haproxy IP
Correct. It's a proxy.

I think there is special configuration in haproxy to preserve the original requestor's ip address, but I couldn't be bothered.
You don't need much for this, for haproxy, just add this:
Code:
option forwardfor header X-Client-IP

On the nginx side you need the HTTP_REALIP option (it's on by default) and two configuration lines:
Code:
    set_real_ip_from 127.0.0.1/32;
    real_ip_header   X-Client-IP;

The set_real_ip_from should be set to the internal address of the haproxy. Any request from that address (or addresses) will have it's 'source' address translated to the address from the X-Client-IP header. The nginx logs (and web applications) will then show the "real" source address instead of the haproxy server's address.
 
Thanks. Now I will "bother" :)

I already had this in haproxy.
Code:
option forwardfor header X-Client-IP

I was just missing the nginx piece.
 
Thanks, but what about the traffic that doesn't go through the proxy (nginx in my case)? While I'm using it to proxy some traffic, ssh just uses a different port than the host and pf redirects everything, completely bypassing nginx.
 
Post your pf.conf, I'm betting you're doing something wrong with NAT.
 
This is what the NAT part looks like:
Code:
ext_if = "vtnet0"
jail_if = "lo1"

# NAT
nat pass on $ext_if inet from $jail_if:network to any -> $ext_if

I'm not sure if NAT can be responsible. If I ssh into a jail from the host itself instead of an external machine, it still looks like the connection is coming from the jail's own IP.
 
I'm not sure if NAT can be responsible.
That actually looks good, so that can't be it.

If I ssh into a jail from the host itself instead of an external machine, it still looks like the connection is coming from the jail's own IP.
I have a very similar setup as yours but my connections come from the host's IP as expected. So I'm wondering what's different.
 
When you don't know what's actually happening with the packets, it's best to use a packet sniffer like tcpdump.
Use this command to show all IP packets with numeric IP addresses:
Bash:
sudo tcpdump -nvvv

Or you can sniff on a specific interface:
Bash:
sudo tcpdump -nvvv -i vtnet0

You should be seeing the packets coming from 192.168.2.1 and not 192.168.2.11.

Also, can you post the output of ifconfig vtnet0 ? I believe vtnet0 should be an epair, right?
 
vtnet0 is the external interface (it's a VPS).

The jails are supposed to use the interface lo1

Using tcpdump(1), I confirmed that the packets do indeed start and end at the same internal IP Address.
This appeared when I accessed the website hosted in the jail with the IP 192.168.2.11
Code:
09:48:08.199830 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->b55d)!)
    192.168.2.11.80 > 192.168.2.11.39149: Flags [.], cksum 0x858d (incorrect -> 0xbdbb), seq 4200, ack 430, win 64516, options [nop,nop,TS val 1984554559 ecr 4232138589], length 0
Yes, every packet's checksum is bad

However, something strange I noticed is that the entire traffic happens on lo0. Despite the jails's IP addresses being assigned to lo1, that interface sees no traffic at all.
Code:
$ iocage get ip4_addr www
lo1|192.168.2.11

This is the entire output of ifconfig:
Code:
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether [MAC address]
    inet [Public IPv4] netmask 0xff000000 broadcast 0.255.255.255
    inet6 fe80::74e0:89ff:fee9:9144%vtnet0 prefixlen 64 scopeid 0x1
    inet6 [IPv6 subnet]::1 prefixlen 64
    inet6 [IPv6 subnet]::1:1 prefixlen 128
    inet6 [IPv6 subnet]::4:1 prefixlen 128
    inet6 [IPv6 subnet]::3:1 prefixlen 128
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 192.168.2.0 netmask 0xffffff00
    # These 4 addresses are configured as aliases
    inet 192.168.2.1 netmask 0xffffffff
    inet 192.168.2.11 netmask 0xffffffff
    inet 192.168.2.14 netmask 0xffffffff
    inet 192.168.2.13 netmask 0xffffffff
    inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
 
Where is this coming from?
Code:
inet 192.168.2.0 netmask 0xffffff00
That's a network address and should never be assigned to a network interface.
 
The jails are supposed to use the interface lo1

Why?

Where is this coming from?
Code:
inet 192.168.2.0 netmask 0xffffff00

I also don't understand that magic :confused:

Why not just use vtnet0/192.168.2.1 as the gateway (defaultrouter) for the jails. That's the automagic of basic LAN IP jails routing provided by the host, without VNET.

Here is my iocage jail "config.json" for simple LAN jail:


Code:
{

"allow_raw_sockets": "1",
"cloned_release": "12.0-RELEASE",
"host_hostname": "vhost",
"host_hostuuid": "vhost",
"ip4_addr": "bge0|10.1.10.55",
"jail_zfs_dataset": "iocage/jails/vhost/data",
"last_started": "2019-09-28 01:03:56",
"release": "12.0-RELEASE-p10",
"sysvmsg": "new"

}

Code:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 18:03:73:1a:35:f7
        inet 10.1.10.5 netmask 0xffffff00 broadcast 10.1.10.255
        inet 10.1.10.55 netmask 0xffffffff broadcast 10.1.10.55
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
vtnet0 is the external interface (it's a VPS).

The jails are supposed to use the interface lo1

Using tcpdump(1), I confirmed that the packets do indeed start and end at the same internal IP Address.
This appeared when I accessed the website hosted in the jail with the IP 192.168.2.11
Code:
09:48:08.199830 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->b55d)!)
    192.168.2.11.80 > 192.168.2.11.39149: Flags [.], cksum 0x858d (incorrect -> 0xbdbb), seq 4200, ack 430, win 64516, options [nop,nop,TS val 1984554559 ecr 4232138589], length 0
Yes, every packet's checksum is bad

However, something strange I noticed is that the entire traffic happens on lo0. Despite the jails's IP addresses being assigned to lo1, that interface sees no traffic at all.
Code:
$ iocage get ip4_addr www
lo1|192.168.2.11

This is the entire output of ifconfig:
Code:
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether [MAC address]
    inet [Public IPv4] netmask 0xff000000 broadcast 0.255.255.255
    inet6 fe80::74e0:89ff:fee9:9144%vtnet0 prefixlen 64 scopeid 0x1
    inet6 [IPv6 subnet]::1 prefixlen 64
    inet6 [IPv6 subnet]::1:1 prefixlen 128
    inet6 [IPv6 subnet]::4:1 prefixlen 128
    inet6 [IPv6 subnet]::3:1 prefixlen 128
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 192.168.2.0 netmask 0xffffff00
    # These 4 addresses are configured as aliases
    inet 192.168.2.1 netmask 0xffffffff
    inet 192.168.2.11 netmask 0xffffffff
    inet 192.168.2.14 netmask 0xffffffff
    inet 192.168.2.13 netmask 0xffffffff
    inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
As SirDice mentioned, you have configured somehow lo1 to have inet 192.168.2.0 netmask 0xffffff00. It's a network IP address and I have no idea what happens if you set it as an address of an interface but it seems wrong.
In my opinion you need to remove this IP address from lo1. Also, if your netmask is 0xffffffff for all the jails, you need to route the packets (have the NAT). Or you could connect them in a single subnet and let them talk directly via netmask 0xffffff00.

In summary, if your IP settings are wrong, there is no need to debug the packets. First fix the IP addresses.
 
Where is this coming from?
Code:
inet 192.168.2.0 netmask 0xffffff00
That's a network address and should never be assigned to a network interface.

I really don't remember why I put that into my rc.conf in the past. Therefore, I removed it and changed 192.168.2.1 to be the that interface's primary address instead of an alias. This didn't change anything about the original problem though.

The actual solution was something else, as suggested by roccobaroccoSC:
Also, if your netmask is 0xffffffff for all the jails, you need to route the packets (have the NAT). Or you could connect them in a single subnet and let them talk directly via netmask 0xffffff00.

In summary, if your IP settings are wrong, there is no need to debug the packets. First fix the IP addresses.

When configuring the IPs for the jails, I had not put the /24 at the end, which resulted in the wrong netmask.
Instead of iocage set ip4_addr="lo1|192.168.2.11/24" www, I had used iocage set ip4_addr="lo1|192.168.2.11" www

After fixing this, my ifconfig looks like this:
Code:
inet 192.168.2.1 netmask 0xffffff00
inet 192.168.2.11 netmask 0xffffff00
inet 192.168.2.14 netmask 0xffffff00
inet 192.168.2.13 netmask 0xffffff00


When speaking to the jails, the host now correctly uses 192.168.2.1, so my problem is solved.

Many thanks to everybody who helped!
 
I really don't remember why I put that into my rc.conf in the past. Therefore, I removed it and changed 192.168.2.1 to be the that interface's primary address instead of an alias. This didn't change anything about the original problem though.

The actual solution was something else, as suggested by roccobaroccoSC:


When configuring the IPs for the jails, I had not put the /24 at the end, which resulted in the wrong netmask.
Instead of iocage set ip4_addr="lo1|192.168.2.11/24" www, I had used iocage set ip4_addr="lo1|192.168.2.11" www

After fixing this, my ifconfig looks like this:
Code:
inet 192.168.2.1 netmask 0xffffff00
inet 192.168.2.11 netmask 0xffffff00
inet 192.168.2.14 netmask 0xffffff00
inet 192.168.2.13 netmask 0xffffff00


When speaking to the jails, the host now correctly uses 192.168.2.1, so my problem is solved.

Many thanks to everybody who helped!
Nice! You could have done it also by putting all jails in separate subnets and routing between them (via gateway_enable=YES) but this cannot be done with a network mask 0xffffffff. You need at least 2 hosts per subnet, so the mask can be at most 0xfffffffc. In this case you also need to make sure that each jail has properly filled routing tables to reach the other jails' subnets.
But putting all jails in one subnet is the simplest and easiest thing to do.
 
Back
Top