How do I tell if somebody has gained access to / tampered with my server?

A

andyzammy

Hi,

After playing with my router in an attempt to open my server up to the outside world, I gave up after failing to ever get an SSH connection via my external IP address. Assuming it was a no go, and the ISP wasn't allowing it, I just left everything "open" as it wasn't working anyway.

Recently, I was reading the root emails because I've never done that and thought why not, and was surprised to find common username login attempts from here, there and everywhere. I guess everyone can see my machine but me!

I wasn't really concerned, because the first thing I did before trying this was to configure key-only access via sshd. I must have forgotten to do that, as I just keyed in through a VM.

I'm going to reinstall everything anyway just in case, but before I do, is there a way I can verify that anything has been messed with? There's nothing in the logs to say anybody got in, but then again, they could have modified the logs. I did notice that a user writable flag was added to /usr/local/bin/sudo, but I'm not sure if that's relevant (I read it in the mail, there were other things but just as meaningless to me).

I've always been curious about this sort of thing, and I guess now I've got a chance to take a look. My data still appears to be there, so there were no malicious wipes or anything. Nothing here is critical stuff, so I'm not really concerned that anybody (might have) got in. What would be useful now though is confirmation. I picked a pretty solid password, but I'm not naive enough to think that was enough.

Any tips, tutorials, jokes are welcome :D
 
andyzammy said:
to find common username login attempts from here, there and everywhere.
Click to expand...
That's normal. Script kitties [sic] are scanning the whole IP range for SSH servers all the time and try to log into common accounts using weak and/or empty passwords. As long as you have no weak passwords (and/or have password authentication disabled in the first place) it's no (immediate) cause for concern. Most of these brats appear too retarded to try other ports than the default port 22, so running sshd on a high-numbered non-standard port might stop most of the logfile cluttering, at the cost of the slight inconvenience of having to use a different port.

Please do note that running sshd on a non-standard port only keeps your logs clean(er) by thwarting the stupidest of wannabes. It does little to deter attackers who have at least some clue of what they are doing.
 
andyzammy said:
I'm going to reinstall everything anyway just in case, but before I do, is there a way I can verify that anything has been messed with?
Click to expand...
An Intrusion Detection System, or IDS, would be a very suitable solution.

Such a program basically builds a database filled with checksums of the files it needs to monitor, after which it can perform checks where it will compare the file's current state with that in the database. If that doesn't match then something has changed.

There are several good IDS environments available, a good way to get an overview is to use this command from /usr/ports: # make search key=ids | grep -A 10 security | less.

Almost forgot: my personal favourite is security/tripwire, also because it encrypts everything, including the so called policy files, which make it harder (but not impossible!) for an intruder to tamper with the IDS itself.
 
fonz said:
That's normal. Script kitties [sic] are scanning the whole IP range for SSH servers all the time and try to log into common accounts using weak and/or empty passwords. As long as you have no weak passwords (and/or have password authentication disabled in the first place) it's no (immediate) cause for concern. Most of these brats appear too retarded to try other ports than the default port 22, so running sshd on a high-numbered non-standard port might stop most of the logfile cluttering, at the cost of the slight inconvenience of having to use a different port.

Please do note that running sshd on a non-standard port only keeps your logs clean(er) by thwarting the stupidest of wannabes. It does little to deter attackers who have at least some clue of what they are doing.
Click to expand...

The usual brute-force attack goes for the low-hanging fruit. They don't bother testing a large range of ports. Unless you're the target of a dedicated break-in attempt, using a non-standard port is quite useful.
 
I get 5-10 SSH brute-force attempts a day because I have chosen to keep the service listening on the standard port 22 and it's open for all addresses. I do have security/sshguard-pf to kick out the knockers very quickly though.
 
If you're really concerned, you want to be sending logs to another machine via syslog, and ensuring that the only open port on that machine is syslog. Bear in mind that if someone has compromised your machine there's every chance they can delete/modify/manufacture logs as they please.

By sending them to a remote machine, this is no longer possible.

If you are running SSH facing the Internet, consider using public key authentication and turning off the ability to log in via passwords.
 
@throAU has a very good advice for you - choose the highest security and the lowest profile/exposure possible. Also, put SSH on a different port, as @fonz suggested. We always run our sshd on ports beyond 10000 and so the logs stay clean.
 
Last edited by a moderator:
kpa said:
I get 5-10 SSH brute-force attempts a day because I have chosen to keep the service listening on the standard port 22 and it's open for all addresses.
Click to expand...
Only 5-10? I get a lot more than that x(

I do have security/sshguard-pf to kick out the knockers very quickly though.
Click to expand...
Yep, it works like a charm. Just install and forget about it. I like how it only blocks addresses for a certain period of time before releasing it again. This works brilliantly if you happen to have fat fingers and type your own password wrong a couple of times. If you get locked out just wait a while for sshguard to release the IP and you can try again.
 
andyzammy said:
After playing with my router in an attempt to open my server up to the outside world, I gave up after failing to ever get an SSH connection via my external IP address. Assuming it was a no go, and the ISP wasn't allowing it, I just left everything "open" as it wasn't working anyway.

Recently, I was reading the root emails because I've never done that and thought why not, and was surprised to find common username login attempts from here, there and everywhere. I guess everyone can see my machine but me!
Click to expand...
I'm guessing you were trying to connect to your external IP address from within your LAN. That's not going to work. Find a friend and use his connection to test it from. The firewall has to NAT the outgoing connection but because the outgoing connection never really leaves the interface the redirection simply never gets used.

My data still appears to be there, so there were no malicious wipes or anything.
Click to expand...
Keep in mind that most attackers are usually not interested in your data. They usually won't remove anything either as that will get noticed immediately by the owner. They want to hold on to the box for as long as possible so they can use it as a "springboard" to attack other systems. Any traces would then lead back to you instead of the original attacker.
 
SirDice said:
Keep in mind that most attackers are usually not interested in your data. They usually won't remove anything either as that will get noticed immediately by the owner. They want to hold on to the box for as long as possible so they can use it as a "springboard" to attack other systems. Any traces would then lead back to you instead of the original attacker.
Click to expand...

This is very true. Typically they'll just add the machine to their botnet and use it to scan others. This is the most common way compromised hosts will get discovered, as that tends to trigger network intrusion detection systems (NIDS), or complaints from those receiving scans.

Another thing to look into is setting your firewall to restrict which hosts or subnets can connect to port 22 (if this is possible within your network). PF and Netfilter on Linux make this pretty easy. I commonly restrict work boxes to the VPN or internal hosts. It's just another layer of defence, but also keeps the logs quiet for easy reading. I'm wary of having ports open to the internet, unless I absolutely have to.
 
The only way to tell if your server has been compromised (AFAIK) is with the following:
  1. See if any of your desktop icons have been moved.
  2. Check if you still have the exact same amount of trash in the recycle bin.
  3. Firefox bookmarks may have been added or removed.
  4. The desktop wallpaper has been changed.
If you notice any of the following, head over to your nearest PC World immediately and seek the help of a PC professional! ;)

Personally I always put sshd on a different port since that pretty much always is enough to prevent these silly scanners and brute force attempts. As a bonus, if I put it on port 443, it means that I can get to my server from overly restrictive corporate firewalls.
 
The Recycle Bin used to be at the bottom-rightmost tile, but now it's at the top left. There must have been an attacker because I sure wasn't the one who resized my desktop...

Thanks for all the suggestions guys. Can installing an IDS really help me at this stage, or are these suggestions for my next re-install? As stated in the first post, I was quite hoping to confirm either way before I reinstalled. Any odd binary names I can look out for, perhaps?

I recall reading the handbook about being able to compare operating systems, which is a bit useless to me in this state as I don't have a "before" snapshot of checksums, but it made me wonder if there was some kind of list kept in /usr/src of file structure, which could hint at some binaries being there that aren't actually part of the base. Also, I know that there's a program that will go through the file system and check flags, and restore them to their intended states if you've accidentally toasted perms (though I can't recall the name). Could anything like this help me?

Again, I realize that someone capable could make it look as though nobody was ever there, so I will reinstall, but I was really just curious if there are any obvious ways to tell if an unprepared system like mine has been broken into.
 
An IDS won't prevent a break-in but, if set up correctly, can be used to find out what happened after the fact. It can also alert you about any on-going attacks.

As for checking the system, you may have heard of security/tripwire. What it basically does is created checksums (hashes) for all your files. You store them at a safe location and compare them from time to time. Any file that's modified would get flagged by it. A few periodic(8) scripts do something similar, they check for the presence of SUID files. A file like /tmp/somerandomname shouldn't have the SUID bit set for example. That would be an indicator something fishy is going on.
 
Fail2ban (security/py-fail2ban) is also an excellent program similar to sshguard which scans log files for other services (not just SSH) such as Apache, Postfix, Dovecot etc., and updates your firewall rules to block attempts.
 
You must log in or register to reply here.
Back
Top