Hi, all. I have a FreeBsd 13.0 server with jails. I use the cbsd utility for configuring jails. There is a problem, that I can't download big files from jails. First time I faced with it on jail with a dovecot server, I often can't download emails with attachments more than 1MB. Downloading failed often and for example file 3MB I can download only from 5-10 trying. I started investigating issue and now I see the same behavior on another jail with nginx. I created 200MB file and try to download it, downloading always fails after different time. Ususally downloading stopped after 0.5...10MB. Wget shows Connection closed or 206 Partial Content. Nginx logs looks like this
In tcpdump I see that downloading stops after the server had sent PUSH and FIN flags.
ipfw rules a follows
Does anybody have any idea, what problem is? Thanks in advance.
Code:
1.2.3.4 - - [01/Jun/2022:17:56:31 +0200] "GET /tst.file HTTP/1.1" 200 6791168 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:56:34 +0200] "GET /tst.file HTTP/1.1" 206 696320 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:56:38 +0200] "GET /tst.file HTTP/1.1" 206 520192 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:56:43 +0200] "GET /tst.file HTTP/1.1" 206 1654784 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:58:07 +0200] "GET /tst.file HTTP/1.1" 200 32768 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:58:12 +0200] "GET /tst.file HTTP/1.1" 200 4403200 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:58:14 +0200] "GET /tst.file HTTP/1.1" 206 69632 "-" "Wget/1.21.2" "-"
1.2.3.4 - - [01/Jun/2022:17:58:15 +0200] "GET /tst.file HTTP/1.1" 200 98304 "-" "Wget/1.21.2" "-"In tcpdump I see that downloading stops after the server had sent PUSH and FIN flags.
Code:
20:52:52.124256 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4753393, win 3653, options [nop,nop,TS val 1233667563 ecr 3937964620], length 0
20:52:52.124345 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4769333:4775125, ack 139, win 1027, options [nop,nop,TS val 3937964660 ecr 1233667563], length 5792: HTTP
20:52:52.124363 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4753393, win 3699, options [nop,nop,TS val 1233667563 ecr 3937964620], length 0
20:52:52.181537 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4757749, win 3631, options [nop,nop,TS val 1233667604 ecr 3937964651], length 0
20:52:52.181704 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4775125:4782365, ack 139, win 1027, options [nop,nop,TS val 3937964711 ecr 1233667604], length 7240: HTTP
20:52:52.181997 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4757749, win 3699, options [nop,nop,TS val 1233667604 ecr 3937964651], length 0
20:52:52.182674 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4759197, win 3677, options [nop,nop,TS val 1233667604 ecr 3937964651], length 0
20:52:52.182756 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4782365:4785261, ack 139, win 1027, options [nop,nop,TS val 3937964711 ecr 1233667604], length 2896: HTTP
20:52:52.182954 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4766437, win 3699, options [nop,nop,TS val 1233667607 ecr 3937964651], length 0
20:52:52.183072 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4785261:4795397, ack 139, win 1027, options [nop,nop,TS val 3937964720 ecr 1233667607], length 10136: HTTP
20:52:52.183292 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4769333, win 3699, options [nop,nop,TS val 1233667608 ecr 3937964660], length 0
20:52:52.183379 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4795397:4798293, ack 139, win 1027, options [nop,nop,TS val 3937964720 ecr 1233667608], length 2896: HTTP
20:52:52.183727 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4772229, win 3699, options [nop,nop,TS val 1233667609 ecr 3937964660], length 0
20:52:52.183816 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4798293:4801189, ack 139, win 1027, options [nop,nop,TS val 3937964720 ecr 1233667609], length 2896: HTTP
20:52:52.184169 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4775125, win 3699, options [nop,nop,TS val 1233667610 ecr 3937964660], length 0
20:52:52.184253 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4801189:4804085, ack 139, win 1027, options [nop,nop,TS val 3937964720 ecr 1233667610], length 2896: HTTP
20:52:52.232359 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4778021, win 3699, options [nop,nop,TS val 1233667657 ecr 3937964711], length 0
20:52:52.232562 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4804085:4806981, ack 139, win 1027, options [nop,nop,TS val 3937964761 ecr 1233667657], length 2896: HTTP
20:52:52.232609 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4780917, win 3699, options [nop,nop,TS val 1233667659 ecr 3937964711], length 0
20:52:52.232686 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4806981:4809877, ack 139, win 1027, options [nop,nop,TS val 3937964761 ecr 1233667659], length 2896: HTTP
20:52:52.232720 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4783813, win 3699, options [nop,nop,TS val 1233667660 ecr 3937964711], length 0
20:52:52.232833 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4809877:4812773, ack 139, win 1027, options [nop,nop,TS val 3937964761 ecr 1233667660], length 2896: HTTP
20:52:52.232870 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4791053, win 3699, options [nop,nop,TS val 1233667661 ecr 3937964711], length 0
20:52:52.232985 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [.], seq 4812773:4820013, ack 139, win 1027, options [nop,nop,TS val 3937964770 ecr 1233667661], length 7240: HTTP
20:52:52.233009 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4795397, win 3699, options [nop,nop,TS val 1233667662 ecr 3937964720], length 0
20:52:52.233073 IP 6.7.8.9.http > 1.2.3.4.57988: Flags [FP.], seq 4820013:4821249, ack 139, win 1027, options [nop,nop,TS val 3937964770 ecr 1233667662], length 1236: HTTP
20:52:52.233091 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4798293, win 3699, options [nop,nop,TS val 1233667665 ecr 3937964720], length 0
20:52:52.233708 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4804085, win 3699, options [nop,nop,TS val 1233667666 ecr 3937964720], length 0
20:52:52.270232 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4806981, win 3699, options [nop,nop,TS val 1233667702 ecr 3937964761], length 0
20:52:52.270388 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4809877, win 3699, options [nop,nop,TS val 1233667702 ecr 3937964761], length 0
20:52:52.270720 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4812773, win 3699, options [nop,nop,TS val 1233667703 ecr 3937964761], length 0
20:52:52.271430 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4818565, win 3699, options [nop,nop,TS val 1233667703 ecr 3937964770], length 0
20:52:52.271882 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [.], ack 4821250, win 3657, options [nop,nop,TS val 1233667704 ecr 3937964770], length 0
20:52:52.920868 IP 1.2.3.4.57988 > 6.7.8.9.http: Flags [F.], seq 139, ack 4821250, win 3699, options [nop,nop,TS val 1233668340 ecr 3937964770], length 0ipfw rules a follows
Code:
00100 18 1726 allow ip from any to any via lo0
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
00400 0 0 deny ip from any to ::1
00500 0 0 deny ip from ::1 to any
00600 11974 715069 nat 1 log ip from any to 6.7.8.9 in via vtnet0
00700 12 1100 nat 2 ip from any to any in via tun0
00800 0 0 nat 3 ip from any to any in via tun1
00900 12 1100 allow ip from any to any via tun0
01000 0 0 allow ip from any to any via tun1
01100 15734 35606560 skipto 50000 log ip from 10.11.0.0/24 to any out via vtnet0 keep-state :default
01200 6 214 allow icmp from any to me icmptypes 0,3,4,8,11
01300 6 214 allow icmp from me to any icmptypes 0,3,4,8,11
01400 0 0 check-state :default
01500 117 10938 allow log ip from me to any keep-state :default
01600 0 0 skipto 50000 ip from 10.10.10.0/24 to any out via vtnet0 keep-state :default
01700 0 0 skipto 50000 ip from 10.10.11.0/24 to any out via vtnet0 keep-state :default
01800 0 0 deny ip from 192.168.0.0/16 to any in via vtnet0
01900 0 0 deny ip from 172.16.0.0/12 to any in via vtnet0
02000 0 0 deny ip from 127.0.0.0/8 to any in via vtnet0
02100 0 0 deny ip from 0.0.0.0/8 to any in via vtnet0
02200 0 0 deny ip from 169.254.0.0/16 to any in via vtnet0
02300 0 0 deny ip from 192.0.2.0/24 to any in via vtnet0
02400 0 0 deny ip from 204.152.64.0/23 to any in via vtnet0
02500 0 0 deny ip from 224.0.0.0/3 to any in via vtnet0
02600 0 0 deny log tcp from any to any 113 in via vtnet0
02700 0 0 deny log ip from any to any frag offset in via vtnet0
02800 106 7564 deny log tcp from any to any established in via vtnet0
02900 5 224 allow tcp from any to me 21,50050-50100
03000 2934 5667261 allow tcp from any to me 22 setup keep-state :default
03100 20 1036 allow tcp from any to me 25
03200 0 0 allow udp from 193.218.142.109 to me 53 keep-state :default
03300 0 0 allow udp from any to me 53 via tun0 keep-state :default
03400 0 0 allow udp from any to me 53 via tun1 keep-state :default
03500 0 0 allow log tcp from any to me 143
03600 37 2180 allow tcp from any to me 80
03700 0 0 allow tcp from any to me 8080
03800 2 84 allow tcp from any to me 443
03900 0 0 allow udp from any to me 123 keep-state :default
04000 0 0 allow tcp from any to me 465
04100 4 276 allow udp from any to me 2012
04200 0 0 allow udp from any to me 5001 keep-state :default
04300 0 0 allow tcp from any to me 5001 keep-state :default
04400 421 20464 deny log ip from any to any
50000 6116 35040733 nat 1 log ip from any to any out via vtnet0
50100 15464 24910904 allow log ip from any to any
51000 0 0 nat 2 ip from any to any out via tun0
51100 0 0 allow ip from any to any
65535 274 37326 allow ip from any to anyDoes anybody have any idea, what problem is? Thanks in advance.
