how do you stop intruders

So you've got intruders, which hijacked your VPS account and could have done whatever to your machine, you are absolutely unsure about it.

The sane approach to resolve this is:

1. Reboot your machine into rescue mode
2. Try to figure out what happened, how they came in and what they have done - if you got a webserver, also pay attention to PHP files there etc. as well, use rkhunter/maldet
3. Delete your VPS and install it from scratch
4. Restore all necessary data from your backup - this means data, but no binaries/shell scripts
 
This could be a genuine input validation ssh bug.
I feel something is amiss with the timeline.

Sadly, it is, and it makes the diagnosis of it being an ssh bug unlikely. Furthermore, the probability of finding ssh bugs by buffer stuffing is infinitesimally small, unless the OP is running an ancient version of FreeBSD with known vulnerabilities.

User Paul attempted to log in from Rogers Cable (in Canada) on 7-Mar at 16:02. I'll assume that this was a real login attempt by the real Paul, with a mistyped password. Unfortunately, afterwards we can't see the real login attempts.

Then on 10-Mar at 1:56, a su succeeded on the first try. We have to assume that this was the real user paul again.

Then on 10-Mar at 3:42..., there was a series of failed su attempts. This was probably the hacker.

Now, the "buffer stuffing" only happened a day later, on 11-Mar at 2:25. The failed su attempts then continued another day later, on 12-Mar at 20:48, followed by a successful su a minute later.

One possibility is that the hacker logged in as paul, and then successfully guessed the root password after a few attempts (perhaps it was "password" or "snoopy", as there were only a handful of attempts).

Another possibility is that the real Paul was simply doing normal system administration, and is not very good at typing and often mistypes the root password; but above, he said that it wasn't him.
 
2. Try to figure out what happened, how they came in and what they have done - if you got a webserver, also pay attention to PHP files there etc. as well, use rkhunter/maldet
For a person with good computer security skills, that might be an interesting thing to do: forensics on how the crime was committed. But that's not the situation we have here, as the OP seems to not be highly experienced.

But even for a knowledgeable and skilled person, the "lessons learned" are not very valuable: figuring out how the hackers came in the last time has little predictive power for how they are going to try next time. To put it allegorically: The barn that is computer security has many many doors; closing one particular barn door doesn't really help much if the others are not managed well and left wide open. The correct advice for the OP is: Install up-to-date versions of software (OS, applications such as web servers and ssh daemons); maintain those versions well; close as many means of access as possible (for example, always disable root ssh); use good keys and other authentication. And most importantly, make yourself smart about good computer security practices.
 
Then on 10-Mar at 1:56, a su succeeded on the first try. We have to assume that this was the real user paul again.
Right. I read that as bad guy but it could have been legit. That would solve the timeline issue.

Bunch of bad guesses, then kex handshake input crack, then paul to root
 
One little problem with that theory.
If you login your console messages would not be stacked like that.

My feeling is 'actual' user paul did not login on the 10th. or messages would not be showing.

I wish we had logs to pick apart..
 
I still have pf setup to rate-limit & blacklist ssh attempts even though I use a key now.
On Linode it is standard hardening.
Praise The Book of Pf for its wisdom......

This event did cause me to review my defensive posture. Lots of noise but nothing bad.
I had to quiet down pf logging. That place is noisy.
 
A little thing I do, in case you always use ssh from the same place, then do not allow the entire internet to connect through your ssh.
You can either allow your static IP(if you have one), or allow only your ISP's subnet (if you have a dynamic one).
The only thing you have to do is to identify what your ISP's subnet is, play with net-mgmt/ipcalc to find a better netmask, and add it to your ssh rule in the firewall.
Then attacks are reduced drastically.
 
Back
Top