PF Issue with configuration blocking access to jails on ip aliases attached to external interface

lifepillar

lifepillar

Hi,
when I activate PF in my server, I cause connectivity to my jails to be blocked (even when it should not). With PF disabled, everything works. I am likely doing something stupid, but I need other eyes to look at it.

My jails are bound to the only active interface igb0:

Code: 
root@host # ifconfig

igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether xx:xx:xx:xx:xx
    inet 10.66.90.100 netmask 0xffffff00 broadcast 10.66.90.255
    inet 10.66.90.101 netmask 0xffffffff broadcast 10.66.90.101
    inet 10.66.90.102 netmask 0xffffffff broadcast 10.66.90.102
    [...]
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
[...]

Here 10.66.90.100 is the host's address in my LAN, and the rest are aliases assigned to the jails. My /etc/rc.conf is:

Code: 
root@host # cat /etc/rc.conf
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="server.home.arpa"
ifconfig_igb0="inet 10.66.90.100 netmask 255.255.255.0"
defaultrouter="10.66.90.1"
sshd_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"

I create jails with Bastille, e.g. the first jail was created with bastille create jail0 13.0-RELEASE 10.66.90.101 igb0. Inside jail0:

Code: 
root@jail0 # ifconfig

igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether xx:xx:xx:xx:xx:xx
    inet 10.66.90.101 netmask 0xffffffff broadcast 10.66.90.101
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
[...]

My host's PF configuration is as follows:

Code: 
root@host # cat /etc/pf.conf

ext_if="igb0"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
block in all
pass out quick modulate state
antispoof for $ext_if inet
pass in inet proto tcp from any to 10.66.90.100 port ssh flags S/SA keep state
pass in inet proto tcp from any to 10.66.90.101 port https flags S/SA keep state

With this, nc -nvz 10.66.90.101 443 from the host (or from anywhere in my network) succeeds, but curl -k https://10.66.90.101 hangs (even curl -k https://127.0.0.1 from inside the jail hangs). I have no problem connecting via SSH, though. If I comment out the antispoof rule, then curl returns curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 54. What am I doing wrong?
 
I have found that the issue depends on the service running in the jail (nginx) forwarding to a local proxy (via a proxy_pass http://127.0.0.1:8888 directive—below I report the relevant nginx config). Adding the following rule to the host's /etc/pf.conf enables such traffic and makes the service functional (10.66.90.101 is the jail's IP address—see above):

Code: 
pass inet proto tcp from 10.66.90.101 to 10.66.90.101 port 8888 flags S/SA keep state

I was under the impression that such traffic would pass through the loopback interface (hence it would not be filtered as per set skip on lo above), but that does not seem the case. Can anyone enlighten me?

I had tested a similar configuration in a VirtualBox machine, and I did not have to add the rule above.

Nginx config in the jail:

Code: 
server {
    listen 443 ssl;
    server_name myservice.home.arpa;

    error_log  /dev/null;
    access_log /dev/null;

    ssl_certificate      /usr/local/etc/nginx/myservice.crt;
    ssl_certificate_key  /usr/local/etc/nginx/myservice.key;

    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
      proxy_pass http://127.0.0.1:8888;

      proxy_set_header Host $host;
      proxy_set_header Connection       $http_connection;
      proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
      proxy_set_header X-Scheme         $scheme;
      proxy_buffering                   off;
    }
}
 
You must log in or register to reply here.
Back
Top