Solved [Solved] PF + Nginx in a Jail Errors when downloading

zodias · Jul 19, 2014

Hi,
I am currently using FreeBSD 10.0-RELEASE-p7. PF redirects http port to www/nginx running in a jail.
www/nginx error log reports:

Code:

2014/07/19 13:11:12 [alert] 54480#0: [b]*2 sendfile() failed (1: Operation not permitted) while sending response to client[/b], client: 93.152.132.145, server: zlatkoasenov.com, request: "GET /FreeBSD/releases/amd64/10.0/Desktop/All/openjdk-7.65.17%2c1.txz HTTP/1.1", host: "www.zlatkoasenov.com", referrer: "http://www.zlatkoasenov.com/FreeBSD/releases/amd64/10.0/Desktop/All/"

and
#pfctl -si
reports:

Code:

No ALTQ support in kernel
ALTQ related functions disabled
Status: Enabled for 8 days 17:47:41           Debug: Urgent

State Table                          Total             Rate
  current entries                        3               
  searches                         3009010            4.0/s
  inserts                            33226            0.0/s
  removals                           33244            0.0/s
Counters
  match                              43215            0.1/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                     28595            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
root@:/usr/home/zodias #

Every time "sendfile()" error is reported the value of state-mismatch field increases.
It happens only when serving files >= 30MB in size.

I tried changing options in /usr/local/etc/nginx/nginx.conf such as:

Code:

tcp_nopush     on;
proxy_max_temp_file_size 0;

I replaced:

Code:

scrub in all

with

Code:

scrub in all fragment reassemble

So far no success.
Any hint is welcome!

zodias · Jul 19, 2014

Re: PF + Nginx in a Jail Errors when large files are downloa

I got help from a friend, running http://www.unix-heaven.org. On his advice I split :

Code:

#rdr pass on $HOST_IF inet proto tcp from any to $HOST_IP port $PORT_HTTPS  -> $JAIL_WEB_PROXY

with

Code:

rdr on $HOST_IF proto  tcp  from any to $HOST_IP port $PORT_HTTP -> $JAIL_WEB_PROXY
pass in quick on $HOST_IF inet proto tcp  from any to $JAIL_WEB_PROXY port $PORT_HTTP flags S/SA keep state

It appears that keep-state rule is not appended by default and must be declared expressly.

Now download is ok.