Abnormally slow Apache24 file transmitting by lo0 interface in FreeBSD 10.3

D

Dmitriy_K

Hi all!

I have a very strange case.
On hardware server has been installed FreeBSD 10.3 + Apache24 (apache24-2.4.18) for a few sites. Web-server works without jail.
Everything works fine, but had to adjust the indexing (mnogosearch), and during testing it was discovered that Apache24 transmit the large files via the local interface lo0 abnormally slow.
What is transmitted to the external clients for a few seconds, by lo0 is transmitted approximately 16-18 minutes!!!

Code: 
# wget http://www.***/common/101015.pdf
Resolving www.*** (www.***)… 127.0.0.1, ::1
Connecting to www.*** (www.***)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11049247 (11M) [application/pdf]
Saving to: «101015.pdf»
101015.pdf  0%[  ]  47,59K  --.-KB/s  remain 18m 42s


Because of this, indexing of large PDF's is impossible, but I'm amazed by such a specific effect.
I tried to disable PF for check - it does not affect on problem. It more looks like a bug of lo0 in FreeBSD 10.3, although bug of Apache24 also not excluded.

I will be very grateful if anyone tell me what to do with such an unexpected problem.


Code: 
% ifconfig
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
  ether 00:25:90:c0:fa:2a
  inet ****** netmask 0xffffff00 broadcast *****.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (100baseTX <full-duplex>)
  status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
  ether 00:25:90:c0:fa:2b
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
I just made additional test. I installed nginx with a simple test website.
I put to this test site the same test PDF file with 11MB size.
I had the opportunity to compare the transfer rate of the test file by two different web servers.
1) It turned out that nginx is great and provides much greater file transfer speeds than apache. ;-)
I got a file transfer speed of about 260 KB/s for nginx and about 10KB/s for apache.
The file transfer speed is very unstable, but finite time to download a file via nginx the same way - 42s for 11MB file.
2) It turned out that the observed anomaly is only due to a bug of work of the local interface lo0 in FreeBSD 10.3


Transfer speed from 10 to 260 KB/s for local lo0 interface!
I am shocked by such problem!
What do I do?


The server has the usual configuration of OS and software, I have about 20 such servers, but under older versions of FreeBSD. They have no such problems.


Code: 
% ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=128 time=0.026 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=128 time=0.019 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=128 time=0.019 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=128 time=0.021 ms
^C
--- 127.0.0.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.019/0.021/0.026/0.003 ms
 
My sysctl.conf
Code: 
kern.ipc.somaxconn=32768
net.inet.tcp.maxtcptw=51200
net.inet.tcp.sendspace=16384
net.inet.tcp.recvspace=8192
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=16384
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.msl=5000
net.inet.icmp.icmplim=2000
net.inet.icmp.icmplim_output=0
net.inet.tcp.icmp_may_rst=0
net.inet.tcp.syncookies=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=30000
net.inet.tcp.sack.enable=1
net.inet.tcp.delayed_ack=1
net.inet.tcp.delacktime=10
net.inet.ip.random_id=1
net.inet.ip.portrange.first=1024
net.inet.ip.portrange.randomized=0
net.inet.tcp.mssdflt=1460
net.inet.ip.ttl=128
net.inet.ip.redirect=0
net.inet.icmp.drop_redirect=1
net.inet.icmp.log_redirect=0
net.inet.ip.rtexpire=2
net.inet.ip.rtminexpire=2
net.inet.ip.rtmaxcache=256
net.inet.tcp.keepidle=30000
net.inet.tcp.keepintvl=10000
net.inet.tcp.ecn.enable=1
net.inet.tcp.cc.algorithm=cubic

I just now try to change net.inet.tcp.cc.algorithm to "newreno".
It does not affect on problem. (Or I must to do reboot to change the algorithm?)
 
I would actually suggest removing pretty much everything (with the possible exception of somaxconn) and see how things go. Most of the time bad performance is caused by tuning too much (or the wrong things).
 
Thanks for the idea!

I found the source of the problem.
It turned out to be the variable "net.inet.tcp.recvspace=8192" in sysctl.conf
When you install it in the default (65536), bandwidth of lo0 disproportionate increases in 60.000 times.
I got data transfer speeds of 575 MB/s and 1.22 GB/s for Apache and Nginx.

What interesting. For FreeBSD 8.4 (and probable older) this effect is absent.
The variable net.inet.tcp.recvspace always positioned as the variable that determines amount of system memory to service the external connections to the server. Based on this, I used it.
It turned out that starting from FreeBSD 9, it works on the local interface lo0.
It would be good to understand that only local or external also? (Somehow I did not notice any harmful consequences, although this variable used by me for a long time.)
And interesting, it's a feature or a bug? IMHO, it is more consistent in its properties to bug.
There is no point of application net.inet.tcp.recvspace and net.inet.tcp.sendspace to local lo0 interface.

Can anyone to explain why happened such changes in behavior of a variable "net.inet.tcp.recvspace"?
Do I need to write a bug report?
 
I'm not familiar with specific details but FreeBSD has been improving steadily over the years and things that used to require manual tuning are now dynamically adjusted by the system itself. FreeBSD generally does a great job tuning itself.
 
FreeBSD is still the system managed by man in contrast to Windows (and soon Linux). That's fine. ;-)
As for the reasons for existing problems, then unfortunately, it cannot be considered as an improvement of the system.

Maybe someone will be able to give an exact explanation.
 
Calomel is not regarded very highly in the BSD community for its accuracy of the information provided. Especially in the OpenBSD community. Caveat Emptor.
 
You must log in or register to reply here.
Back
Top