Supposedly a loopback problem in a jail

jazzlover · Nov 28, 2020

I have problem with wordpress (WP) REST API in jail. Site Health Status reports two problems, both related to loopback requests, though quite non-specific:

The REST API encountered an error: The REST API request failed due to an error. Error: Connection refused (http_request_failed)
Your site could not complete a loopback request: The loopback request to your site failed, this means features relying on them are not currently working as expected. Error: Connection refused (http_request_failed)

I use single site WP FEMP setup on 12.1-RELEASE-p9 behind www/nginx proxy, which terminates https requests (also 301 redirect all requests to https) and passing them to another www/nginx servers (in jails). In jails I have only local IPs (10.1.1.1, etc) and connections from external IP redirected with pf to proxy (which distribute them to different services, including the WP installation). All usually works smoothly without any problem, except the REST API.

Eventually I managed to pin down the problem to some extent.

If I specify the domain of the WP website in hosts file, like 10.1.1.2 mysite.com, where the IP is for nginx proxy, Site Health Status stops to complain. But other issues appear.
If I specify tor the localhosts the IP of the WP installation, which probably is intended to work this way, the REST API still doesn't work, though I easily access website and REST API from curl from inside the jail.

The wp-cli works except the cron command.
In rc.conf I have:

Code:

syslogd_flags="-ss"
pf_enable="YES"
gateway_enable="YES"
hostname="cloudserver"
ifconfig_vtnet0="inet 123.456.789.123 netmask 255.255.252.0 -lro -tso"
defaultrouter="123.456.789.123"
cloned_interfaces="lo1"
ipv4_addrs_lo1="10.1.1.1-14/28"
jail_enable=YES
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"
blacklistd_enable="YES"
fail2ban_enable="YES"
jail_sysvipc_allow="YES"

The jail.conf (all jails the same):

Code:

exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
mount.fstab = "/etc/fstab.${name}";

host.hostname="$name.myservername";
path="/jails/$name";
ip4.addr = lo1|10.1.1.$ip;
interface = lo1;
#jid="10$ip";
#$sl="-1";

mariadb {
$ip="1";
sysvshm="new";
#securelevel="-1";
#securelevel=$sl;
}
...

The pf.conf

Code:

##### 1. Macros
# Public IP address
ip_pub="123.456.789.101"
# External network interface
if_ext="vtnet0"
sql_port="3306"
ng_pr="10.1.1.2"
mds="10.1.1.1"
shru="10.1.1.4"
red="10.1.1.6"
dns="10.1.1.11"
###### 4. Packet normalization
scrub in on $if_ext no-df random-id
###### 6. Translation
# Allow outbound connections from within the jails
nat on $if_ext from lo1:network to any -> ($if_ext)
##### 7. Redirection
# nginx as reverse proxy jail
rdr pass on $if_ext proto tcp from any to $ip_pub port { http, https } -> $ng_pr
##### 8. Packet filtering
anchor "blacklistd/*" in on $if_ext
anchor "f2b/*"
block in
# Fix jails loopback problem
pass quick on lo0 all
# Filtering loopback traffic in jails
pass quick from $ng_pr to $ng_pr
pass quick from $mds to $mds
pass quick from $shru to $shru
pass quick from $red to $red
pass quick from $dns to $dns
pass out keep state
########### Allow access to external IP
# Allow access to the nginx proxy
pass in on $if_ext proto tcp to $ng_pr port { http, https, $matrix_fed } keep state
pass quick proto tcp from $shru to $ng_pr port { http, https } keep state
########### Allow access between jails
# Allow access local DNS cache server
pass quick proto { tcp, udp } from { lo0, lo1 } to $dns port 53 keep state
# Allow nginx proxy trafic to jails
pass quick proto tcp from $ng_pr to $shru port { http, 81, 8080, 8081 } keep state
# Allow access to databases from jails
pass quick proto tcp from { $ng_pr, $shru } to $mds port $sql_port keep state
# Allow access to redis server from jails
pass quick proto tcp from { $ng_pr, $shru } to $red port { $red_port } keep state

ifconfig in the jail

Code:

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether ca:b7:ac:2d:25:79
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.1.1.4 netmask 0xffffffff
        groups: lo

Some curl(1) output from inside the jail (401 is expected since the endpoint requires authorization):

Code:

# jexec myjail curl -vv localhost/wp-json/wp/v2/plugins
*   Trying 10.1.1.4:80...
* Connected to localhost (10.1.1.4) port 80 (#0)
> GET /wp-json/wp/v2/plugins HTTP/1.1
> Host: localhost
> User-Agent: curl/7.72.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Sat, 28 Nov 2020 19:52:47 GMT
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Robots-Tag: noindex
< Link: <http://mysite.com/wp-json/>; rel="https://api.w.org/"
< X-Content-Type-Options: nosniff
< Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages, Link
< Access-Control-Allow-Headers: Authorization, X-WP-Nonce, Content-Disposition, Content-MD5, Content-Type
< Vary: Origin
< X-Frame-Options: SAMEORIGIN
<
* Connection #0 to host localhost left intact
{"code":"rest_cannot_view_plugins","message":"\u041a \u0441\u043e\u0436\u0430\u043b\u0435\u043d\u0438\u044e, \u0432\u044b \u043d\u0435 \u0438\u043c\u0435\u0435\u0442\u0435 \u043f\u0440\u0430\u0432\u0430 \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u0442\u044c \u043f\u043b\u0430\u0433\u0438\u043d\u0430\u043c\u0438 \u0434\u043b\u044f \u044d\u0442\u043e\u0433\u043e \u0441\u0430\u0439\u0442\u0430.","data":{"status":401}}

I had asked for support on the wordpress.org, but they said that I should solve the problem with loopback in a jail. So, here I am asking for help with loopback problem (if it's the case).
Basically, I would appreciate help with the following questions.
The most important:

How do I ensure that loopback in the jail working as it should?
How I diagnose that it doesn't works as it should for a PHP application?

An additional:

How to test the loopback with PHP code / script / whatever? To emulate PHP request to loopback

Supposedly a loopback problem in a jail

jazzlover