SQUID problem with unavailability of Google services

[mrrc]

This week, when connecting users through a proxy server, some Google services became inaccessible, such as Calendar, Translator, user profile.

When clicking on the services section in the browser on the Google portal, the page does not open and then a connection error is displayed. When directly going to the calendar section, the connection also hangs for a long time without loading the page. At the same time, the Google home page, mail, search work.

Transparent proxying is not used.
Viewing the proxy server logs did not add any understanding, all requests are processed correctly and no errors or prohibitions are observed. There are no other problems with the unavailability of any sites.

When connecting directly (bypassing the proxy server), all Google services work completely correctly.
The platform on which the problem was suddenly discovered:
FreeBSD 13.2-RELEASE-p9
Squid 6.6

A new separate server was deployed for objectivity and finding the cause, but the problem was also reproduced there, its platform.
FreeBSD 13.4-RELEASE-p2
Squid 6.10

I repeat, the problem reproduced suddenly, no changes were made to the proxy server configuration on our side, no problems with Google have arisen for many years. What should I pay attention to in the Squid configuration? Any ideas?

 

[mrrc]

I still can't figure out the reasons, I'm desperate.
I tried using the default configuration file (recommended minimum configuration) to eliminate the problem in my working squid.conf, but the problem remained.
I ask for opinions and help on where to look.

I will be incorrect if I do not say that there are entries in the cache.log, although the IP does not resolve directly to google subdomains, but according to whois, this is the Google LLC farm.

2024/12/21 21:54:57 kid1| conn43356657 local=MYREALIP:53130 remote=142.250.186.142:443 HIER_DIRECT FD 121 flags=1: read/write failure: (60) Operation timed out
current master transaction: master13542083
2024/12/21 21:58:29 kid1| conn43398624 local=MYREALIP:58390 remote=142.250.185.238:443 HIER_DIRECT FD 96 flags=1: read/write failure: (60) Operation timed out
current master transaction: master13553287
2024/12/21 21:58:30 kid1| conn43398801 local=MYREALIP:58419 remote=172.217.16.206:443 HIER_DIRECT FD 898 flags=1: read/write failure: (60) Operation timed out

# lsof -d14
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
syslogd 1606 root 14w VREG 0,121 374327 365383694 /var/log/security (/dev/ada0p5)
named 1636 bind 14u KQUEUE 0xfffff80002267e00 0t0 count=0, state=0x2
squid 1730 squid 14u IPv4 0xfffffe0131c9c000 0 TCP MYDOMAINNAME:62696->lh-in-f102.1e100.net:https (ESTABLISHED)

I've seen similar messages in cache.log before, but the problem with Google surfaced the other day.

 

[dark.initr0]

Hello!
Maybe some thoughts will be helpful.

Can you turn off caching in Squid?
Maybe problem in SSL certificates?

Operation timed out is like some sort of broken connections.
You can try to catch traffic with Wireshark (TCP + UDP + ICMP) and try to analyze but it could be hard task.

 

[mrrc]

cache deny all
no_cache deny all
# ls -la /var/squid/cache/
drwxr-x--- 2 squid squid 512 Dec 21 14:09 .
drwxrwx--- 3 squid squid 512 Nov 1 15:32 ..
-rw-r----- 1 squid squid 123139 Dec 21 23:38 netdb.state

Maybe problem in SSL certificates?

Which ones?
Calendar and Translator are not working.

 

[richardtoohey2]

Something must have changed and if you didn’t change anything on the machine plus a new setup (with newer FreeBSD and Squid) shows exactly the same issue then points to network or third party issues or changes.

Could you try the same setup on Linux? You would expect that NOT to work.

Google changes things quite frequently so if you haven’t changed anything at all then I’d suspect changes at their end. But you say direct connections are working, so that would point to something else on the proxy. DNS? SSL expiry?

 

[astyle]

I will be incorrect if I do not say that there are entries in the cache.log, although the IP does not resolve directly to google subdomains, but according to whois, this is the Google LLC farm.

Try figuring out where that Google farm is physically. It should have a publicly available address. A good way to do it is to look around on netcraft.com. Depending on the geographic location of that server farm, it should be possible to draw some informed ideas about problems stemming from a 3rd party involvement.

 

[mrrc]

Something must have changed and if you didn’t change anything on the machine plus a new setup (with newer FreeBSD and Squid) shows exactly the same issue then points to network or third party issues or changes.

Could you try the same setup on Linux? You would expect that NOT to work.

I also find all this incomprehensible. I don't use Linux in principle, for the sake of objectivity, I just raised a second server with the latest software.

But you say direct connections are working, so that would point to something else on the proxy. DNS? SSL expiry?

Yes, when organizing direct access to *.google.com The services are working properly from the workplace. What could be wrong with DNS, SSL?

Try figuring out where that Google farm is physically. It should have a publicly available address. A good way to do it is to look around on netcraft.com. Depending on the geographic location of that server farm, it should be possible to draw some informed ideas about problems stemming from a 3rd party involvement.

If we analyze the above Google LLC nodes from cache.log, then in the USA.

 

[mrrc]

The reason and solution were not simple and obvious at first glance.
I have two providers accessing the gateway, the main and backup channels, and automatic switching is configured when the connection on the main channel is lost.
To check, I switched the proxy server to the second channel and the problem with partial unavailability of Google services was solved.

I returned it back, used a simple formula in the configuration file with subsequent partial adjustment of ipfw.

# Google via ISP2
acl google dstdomain .google.com
tcp_outgoing_address REAL_IP_ISP2 google

 

[richardtoohey2]

also find all this incomprehensible. I don't use Linux in principle, for the sake of objectivity, I just raised a second server with the latest software.

Just trying to help, but it obviously was wasted time.

DNS is often a cause of unexpected networking issues.

SSL certificates expire and you seemed to have some issues where “nothing had changed” so maybe something had expired. I don’t know how you’ve got things set-up.

Using another OS like Linux helps eliminate FreeBSD as the issue.

You asked for ideas and only you have access to all your systems and networks.

 

[mrrc]

Just trying to help, but it obviously was wasted time.

Any constructive help and attention is welcome.

SSL certificates expire and you seemed to have some issues where “nothing had changed” so maybe something had expired.

What could have expired in the system with software installed yesterday?))
In our country (and maybe not only in our country) there are oddities with access to various Google subdomains from different telecom operators. Apparently, to some extent, this is the case.

 

[astyle]

Just trying to help, but it obviously was wasted time.

Maybe we can try to be nice to each other at least on the Forums, especially in this time of heightened tensions in global politics?

It can be difficult for a specialist to connect the dots between global politics and difficulties in a given field of expertise.