Solved Network requests initiated by jails take a lot of time to be established but are fast afterward

reddy

reddy

We have an annoying issue that we have started to notice a couple of weeks ago, probably at a time when the network was under heavy use, but the issue has not disappeared since them. Maybe the issue has always been there but we didn't notice until recently because we deployed additional services using the network in a more intensive manner.

Basically, external visitors can connect to the server without problem (for example to fetch web pages we host). Similarly, network request initiated from the host (for example using the fetch command) execute rapidly. Jail-to-jail communications do not have any issue as well. However, all external network requests initiated from a jail take a lot of time to start, but after the connection has been established, the download speed is fast. For example, trying to fetch google.com is instant from the host, but the connection may take 10-30 seconds to be established from a jail and sometimes it even times out. I have just tried now from a jail and here is what I get (there is currently essentially no load on the server) :

Code: 
root@web:/home/test # fetch https://www.google.com
fetch: https://www.google.com: Operation timed out
root@web:/home/test # fetch https://www.google.com
fetch: https://www.google.com: Operation timed out
root@web:/home/test # fetch https://www.google.com
fetch: https://www.google.com: size of remote file is not known
www.google.com                                          13 kB   56 MBps    00s

Here is what ifconfig returns for this machine:

Code: 
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet EXTERNAL_IP_1 netmask 0xffffffff broadcast EXTERNAL_IP_1
        inet EXTERNAL_IP_2 netmask 0xffffffff broadcast EXTERNAL_IP_2
        inet EXTERNAL_IP_3 netmask 0xfffffc00 broadcast EXTERNAL_IP_3
        inet 192.168.184.1 netmask 0xffffffff broadcast 192.168.184.1
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.185.1 netmask 0xffffff00
        inet 192.168.185.4 netmask 0xffffff00
        inet 192.168.185.2 netmask 0xffffff00
        inet 192.168.185.3 netmask 0xffffff00
        inet 192.168.185.5 netmask 0xffffff00
        inet 192.168.185.12 netmask 0xffffff00
        inet 192.168.185.13 netmask 0xffffff00
        inet 192.168.185.15 netmask 0xffffff00
        inet 192.168.185.16 netmask 0xffffff00
        inet 192.168.185.17 netmask 0xffffff00
        inet 192.168.185.18 netmask 0xffffff00
        inet 192.168.185.19 netmask 0xffffff00
        inet 192.168.185.20 netmask 0xffffff00
        inet 192.168.185.21 netmask 0xffffff00
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog

And here is the relevant section of /etc/rc.conf

Code: 
#Clone loopback interface for inter-jail communications
cloned_interfaces="lo1"
ifconfig_lo1="inet 192.168.185.1/24"
ifconfig_lo1_alias0="inet 192.168.185.4/24"
ifconfig_lo1_alias1="inet 192.168.185.2/24"
ifconfig_lo1_alias2="inet 192.168.185.3/24"
ifconfig_lo1_alias3="inet 192.168.185.5/24"
ifconfig_lo1_alias4="inet 192.168.185.12/24"
ifconfig_lo1_alias5="inet 192.168.185.13/24"
ifconfig_lo1_alias6="inet 192.168.185.15/24"
ifconfig_lo1_alias7="inet 192.168.185.16/24"
ifconfig_lo1_alias8="inet 192.168.185.17/24"
ifconfig_lo1_alias9="inet 192.168.185.18/24"
ifconfig_lo1_alias10="inet 192.168.185.19/24"
ifconfig_lo1_alias11="inet 192.168.185.20/24"
ifconfig_lo1_alias12="inet 192.168.185.21/24"

#Firewall
pf_enable="YES"
pflog_enable="YES"

#Enable packet forwarding (NAT) for IPV4 and IPV6
gateway_enable="YES"
ipv6_gateway_enable="YES"

And in case it is useful, here are the relevant sections of /etc/pf.conf:

Code: 
ext_if = "vtnet0"
jails_if = "lo1"
jailsnet = "{" $jails_if:network 192.168.184.1 "}"

external1 = "EXTERNAL_IP_1"
external2 = "EXTERNAL_IP_2"
external3 = "EXTERNAL_IP_3"

set skip on lo0
set skip on $jails_if

#Reassemble fragmented packets and drop invalid ones that could be attacks
scrub in all

# A bunch of port mapping instructions go here redirecting outside requests received on external IPs to jails based on the port number used

#Mapping of public IP addresses
nat on $ext_if inet proto tcp from $some_service_jail to any -> $external3
nat on $ext_if inet proto tcp from $mail_jail to any -> $external1
nat on $ext_if inet proto { udp, tcp } from $service5_jail to any -> $external1

nat on $ext_if from $jailsnet to any -> ($ext_if)

pass in quick from $jailsnet to any keep state

#Reject packets coming from spoofed ip addresses and impossible routes
antispoof for $ext_if
antispoof for $jails_if

#Block all by default
block all
pass out all keep state

#Allow ICMP traffic
pass inet proto icmp from any to any

pass from { $jails_if, $jailsnet } to any keep state

pass in inet proto tcp to $ext_if port ssh

Any idea would be greatly appreciated.
 
reddy said:
However, all external network requests initiated from a jail take a lot of time to start, but after the connection has been established, the download speed is fast. For example, trying to fetch google.com is instant from the host, but the connection may take 10-30 seconds to be established from a jail and sometimes it even times out.
Click to expand...
You probably want to check for name resolving issues. This sounds like your primary, and probably the secondary, DNS server are timing out most of the time. Do you get this same delay with ping www.google.com and ping 8.8.8.8 for example?

Code: 
ipv6_gateway_enable="YES"
Why are you enabling routing of IPv6 but have no IPv6 configured anywhere?
 
SirDice said:
You probably want to check for name resolving issues. This sounds like your primary, and probably the secondary, DNS server are timing out most of the time. Do you get this same delay with ping www.google.com and ping 8.8.8.8 for example?

Code: 
ipv6_gateway_enable="YES"
Why are you enabling routing of IPv6 but have no IPv6 configured anywhere?
Click to expand...
Thanks a lot for your help! Unfortunately I am getting the same delays with both www.google.com and 8.8.8.8 so i guess this is not a DNS issue. Actually the network card has an external IPV6 address but I removed it from the post (I removed many details for privacy reasons but I probably should have replaced the IPV6 address with a placeholder for clarity)

EDIT: Actually my test was flawed, I'll get back with the actual results in a moment. Since raw sockets are disabled in the jails I used fetch, but fetch does not work with the 8.8.8.8 IP address even from the host (where this issue is not present). So I guess I will probably enable raw sockets temporarily at least.
 
Yeah, check if there's a difference between connecting to a hostname or an IP address. If connecting to a hostname has a delay but a connection to an IP address does not, then it's likely a DNS issue.

reddy said:
Actually the network card has an external IPV6 address but I removed it from the post (I removed many details for privacy reasons but I probably should have replaced the IPV6 address with a placeholder for clarity)
Click to expand...
Ok, sure, that's understandable. Do your jails have an IPv6 address too? Is that working correctly? You could get a delay when the connection tries IPv6 first, failing, and it falls back to IPv4. This could cause an initial delay on making the connection too.
 
SirDice said:
Yeah, check if there's a difference between connecting to a hostname or an IP address. If connecting to a hostname has a delay but a connection to an IP address does not, then it's likely a DNS issue.


Ok, sure, that's understandable. Do your jails have an IPv6 address too? Is that working correctly? You could get a delay when the connection tries IPv6 first, failing, and it falls back to IPv4. This could cause an initial delay on making the connection too.
Click to expand...

No the jails do not have any IPv6 address, do you think I should add one? However I remembered that I allowed raw sockets in one of the jails a long time ago (the one called service5 in the pf rules above) so I have just used it to save time and pinging and fetching consistently works on it: I can run the fetch www.google.com command 20 times and it will return instantly everytime with the result, also the pings to www.google.com have a 3ms latency, and the latency is the same when pinging 8.8.8.8.

However the big surprise is that when enabling raw sockets in other jails, the problem persists: sometimes the ping or fetch command will work, other times it will just stale waiting to establish the initial connection. It is seemingly random: one time the command will succeed, the next time it will fail (stale indefinitely for ping, or timeout for fetch, or have a long delay before the initial connection is established. If I check what is special with the first jail, I can see the following pf rules:

Code: 
nat on $ext_if inet proto { udp, tcp } from $service5_jail to any -> $external1
nat on $ext_if from $jailsnet to any -> ($ext_if)

proto {udp, tcp} is explicitly mentioned for the service5_jail while it isn't for other jails. However, aren't they included by default if the proto fragment is omitted?

Edit: I have tried adding this parameter to the last rule but it didn't solve the problem.
 
Apparently replacing "($ext_if)" with $external1 in the second rule has solved the issue. Back in the days when I wrote this rule, for some reason I wanted jail initiated requests to use the external IPs in a round robin fashion, but this may have caused some issues with NAT (and it provided exactly zero business value). So I guess the fact that the issue seemed random was that it was picking a random external IP on the interface for each request, but even with this theory I failed to see under which circumstances NAT would have difficulty to route the reply back. Maybe this has to do with IPV6 since the value of $ext_if is set to "vnet0" so maybe the external IPv6 address was occasionally used to send out packets from jails, but even then I am not sure why it would struggle to route them back since PF would store the state just the same and it seems unlikely that it would be confused by IPv6-to-IPv4 conversion. Oh, maybe different external IP addresses were sometimes being used during the handshake which prevented the connection from being established.

Anyway, moving forward I will simply create one jail per service and explicitly choose a specific external IP for every jail. This will make things simpler and better. I have already noticed that our applications hosted on this server run considerably faster when it comes to sending email confirmations and such.
 
reddy said:
Apparently replacing "($ext_if)" with $external1 in the second rule has solved the issue.
Click to expand...
Ah. That interface has multiple addresses, including a private range address. The ($ext_if) would round-robin those addresses. I would remove that private range address from that interface, or use ($ext_if:0) to specifically always take the primary address of the interface.
 
You must log in or register to reply here.
Back
Top