Limiting closed port rst response...

How to locate the IP that's been banging on our web server in the wee hours once every 2-3 days?

About a week ago, I began receiving "Limiting closed port rst response from 261 to 200 packets per second" messages in top and /var/log /messages that appeared (mostly) early in the morning. As I understand it, these alerts are simply telling me that BSD's doing its job. Still, I'd like to find and eliminate whoever it is that's banging on ports.

Based on recommendations I've read on other threads, I've been running tcpdump for the past coupla days, but not seeing anything obvious. To be honest, I'm not sure what I'm looking for?

Any suggestions on how to identify the offending IP so I can block them?
Thanks in advance.
 
As an addendum to my initial message, I found this in another Unix forum and wanted to get the opinion of a more knowledgable BSD'r before I tried it:

If you are running a freebsd server you might be seeing that in your logs an aweful lot. It's an indicator of being port scanned. I found some great advice today on how to stop it and it worked very well.
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50
Add those 3 lines to your /etc/sysctl.conf
They pretty much just blackhole the packets being sent from port scans and the icmp line limits the ping packets people can send.
 
This is the result of tcpdump after one hour. Seems like an unusually high number of dropped packets to me:
 

Attachments

  • tcpdump.jpg
    tcpdump.jpg
    79.2 KB · Views: 923
While tail'ing the server's apache access_log, the message popped-up again.
Should I presume that the comcast IP immediately preceding the two Limiting closed port messages to be the likely culprit? If not, any other suggestions to locate him?
 

Attachments

  • tail.jpg
    tail.jpg
    123.5 KB · Views: 407
Should I presume that the comcast IP immediately preceding the two Limiting closed port messages to be the likely culprit?
They're likely unrelated.

Remember a RST is sent when a SYN is received on a closed port. Your Apache access log shows accepted connections, which is the exact opposite.

If not, any other suggestions to locate him?
Try running tcpdump -n -v 'tcp[tcpflags] & (tcp-rst) != 0'. This will show you all packets that have the RST flag set. Look closely at the destination addresses (the source address should be yours). Then check the source port, that's the port "they" try to connect on and what elicited that RST response.
 
It should be noted that there's very little you can do about it, so it might not be worth spending more time investigating the issue. In particular, you cannot prevent someone from sending SYN packets to arbitrary ports on your server or network.

The only thing you can do is prevent your server from sending RST packets to the culprit at all, saving you a little outgoing bandwidth (and making port scans somewhat slower), but FreeBSD is limiting the RST packets to 200 Hz anyway (as the kernel is telling you), and 200 RST packets per second is close to nothing, unless you're behind a 56k dial-up link.

By the way, you can adjust the limit with the sysctl net.inet.icmp.icmplim. That's a global setting for limits on various kinds of ICMP and RST responses. For more fine-grained control you'll have to use a packet filter (for example IPFW + dummynet).
 
Try running tcpdump -n -v 'tcp[tcpflags] & (tcp-rst) != 0'. This will show you all packets that have the RST flag set. Look closely at the destination addresses (the source address should be yours). Then check the source port, that's the port "they" try to connect on and what elicited that RST response.

I've been running tcpdump (as per above) for several minutes and neither of the two external IPs are ours. The 66 is a US IP owned by Charter, and the 46 is a Ukrainian (which is the country I'm suspecting) IP. Beyond that, I'm not sure how to process what I'm seeing?

UPDATE: Another IP just popped-up: 88.175.2.61 at port 49216, but there's only one entry for that IP....along with several more of the 66 IP following..
 

Attachments

  • tcpdump2.jpg
    tcpdump2.jpg
    170 KB · Views: 414
It should be noted that there's very little you can do about it, so it might not be worth spending more time investigating the issue. In particular, you cannot prevent someone from sending SYN packets to arbitrary ports on your server or network.

I follow what you're saying and it's only because I've never seen that error before in the 20+ years we've been running the server. That said, because we host some popular sites, we've been a target of hackers since day-one, but none've ever been anything more than a nuisance. However, I reckon that's because whoever was maintaining the server (now, me) always investigated/dealt with this kinda weird stuff before it escalated into a problem. But, again, point taken.

The only thing you can do is prevent your server from sending RST packets to the culprit at all, saving you a little outgoing bandwidth (and making port scans somewhat slower), but FreeBSD is limiting the RST packets to 200 Hz anyway (as the kernel is telling you), and 200 RST packets per second is close to nothing, unless you're behind a 56k dial-up link.

Does this relate to these mods I posed previously?
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50
Add those 3 lines to your /etc/sysctl.conf
 
I follow what you're saying and it's only because I've never seen that error before in the 20+ years we've been running the server.
Actually it's not an “error”. It's just an information. I get those regularly, too, and I simply ignore them.
That said, because we host some popular sites, we've been a target of hackers since day-one, but none've ever been anything more than a nuisance. However, I reckon that's because whoever was maintaining the server (now, me) always investigated/dealt with this kinda weird stuff before it escalated into a problem. But, again, point taken.
Code:
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50
Does this relate to these mods I posed previously?
Yes, please refer to the blackhole(4) manual page. As I wrote earlier, disabling RST responses will make port scans somewhat slower. Personally I don't think that it improves security, it's rather a kind of “security by obscurity”. If you don't have vulnerable services on your machine, port scans don't matter anyway. And if you do have vulnerable services, attackers will find them – it just might take them a little longer (but not much) if you use the blackhole feature.

It should also be noted that the blackhole feature can make diagnosing network-related problems more difficult. For example, traceroute(8) (for debugging routing issues) doesn't work anymore. Also, the user experience will suffer in certain circumstances. For example, if there's a problem, it's better to get a useful error message right away, instead of having to wait for an unresponsive browser window, not knowing what's going on.
 
As I wrote earlier, disabling RST responses will make port scans somewhat slower. Personally I don't think that it improves security, it's rather a kind of “security by obscurity”. If you don't have vulnerable services on your machine, port scans don't matter anyway. And if you do have vulnerable services, attackers will find them – it just might take them a little longer (but not much) if you use the blackhole feature.
It should also be noted that the blackhole feature can make diagnosing network-related problems more difficult. For example, traceroute(8) (for debugging routing issues) doesn't work anymore. Also, the user experience will suffer in certain circumstances. For example, if there's a problem, it's better to get a useful error message right away, instead of having to wait for an unresponsive browser window, not knowing what's going on.

Makes sense. Thank you.
 
Back
Top