Squid very slow on FreeBSD 13.1, CPU 100%

O

OlivierW

Hello,

I have run Squid on FreeBSD since FreeBSD 10.x (not sure of the exact minor version) and everything was working perfectly well. During all this years, I have regularly updated Squid and FreeBSD.

Currently I am using Squid 5.7 on FreeBSD 13.1-RELEASE-p5 but had the same problem with Squid 5.6.

Around July 2022, I have updated from FreeBSD 13.0 to 13.1.

And since this upgrade, when many requests over plain HTTP are made from my clients, Squid use 100% of my CPU. It's only using a single core, because I don't have SMP enabled on Squid.

And this CPU load really is noticeable because the web browsing experience is very degraded and slow.

A single computer can generate this load, a very common of HTTP requests causing this problems are:
- Windows 10 updates
- FreeBSD OS updates with "freebsd-update"
- FreeBSD packages updates with "pkg"

My CPU isn't very powerful, it's an Intel Atom N2800, but I have used it for the past 12 years without any problem. First it was with Debian Linux, then FreeBSD since 10.x.

Are there any users with this problem?

What could I search for to find out what is the problem?

Thanks in advance!
 
Thanks cracauer@!

From this documentation, I guess I should run:
Code: 
dtrace -x ustackframes=100 -n 'profile-99 /execname == "squid" && arg1/ {
    @[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks

Right?

At the moment I don't have any Windows 10 user and all my FreeBSD are up to date.
I'll try to reproduce the problem and generate a flame graph as soon as possible.
 
Hello,
I've attached a Flame graph (svg), created when I had the problem a few hours ago.
It happened during a FreeBSD packages update and this time, Squid used "only" around 90% CPU usage.

Not sure what to do with this file. Can anyone help me?
 

Attachments

  • out.zip
    21.4 KB · Views: 84
You spend you CPU time in regular expression matching.

Regex matching is called by access control lists in IPCacheLookupForwarder, whatever that is. There seems to be something screwy going on with regex-driven filters of some kind, or load balancing? Not sure why that would happen a lot during retrieval of single URLs, but that's what it says.

Can you post your config file?
 
complex regexes and also case insensitive ones are a lot slower in 13.x than in 12.x (which uses gnu implementation)
grep -i nxzcvnewerewweladsfjlds /usr/share/dict/web2
is 3x slower on a 13.1 box than on a 12.3 box
as you make the search string longer it goes even worse

the 13.x box cpu is 10% slower than 12.x (as of openssl speed md5)
 
Thanks cracauer@!
It's strange, in the past I was using a lot of "url_regex" rules in Squid, but I removed them about 1 year ago, when the support for PCRE was removed (see the last 4.15 here: https://www.freshports.org/www/squid/ ).

Well, as soon as I have access to my squid machine, I'll share my configuration.


Thanks covacat for this very important information.
Do we have information regarding this problem? Some bug reports to follow?
Does that means regexes are also slower in other tools like PHP or nginx on FreeBSD 13.1?
 
Hmmm, I can't reproduce this using 14-current and textproc/gnugrep

If anything BSD grep takes a bit less CPU time.

Some of the test cases also use strings that are not regular expressions. They shouldn't be affected by slow regex engines.
 
This is confusing. The squid source code schlepps a GNU regex around.

./configure says:
checking if GNUregex needs to be compiled... no

However, it then happily builds and uses the GNU regex in its sources.

Meanwhile ldd shows it linked against the system libregex.
 
Sorry, I've been very late, but thanks for all the answers!

Thanks covacat for the bug report. It's interesting but strange that it mentions FreeBSD 13.0-RC4, while I got this problem only from 13.1.

cracauer@, I have finally found why my Squid was still spending a lot of time dealing with regexes…
In one of my included config file, I still had this lines:
acl zeustracker_denied_domains url_regex -i "/usr/local/etc/squid/zeustracker_domains" http_access deny zeustracker_denied_domains
which I added many years ago to try to block the ZeuS trojan horse. My "zeustracker_domains" was 486 lines.

So, while I'm very happy that my Squid is running fast again, I hope the FreeBSD's libregex problem get fixed soon, hopefully for FreeBSD 14.0. Because it's not fixed in FreeBSD 13.2.
 
You must log in or register to reply here.
Back
Top