Suddenly High traffic on server(hacked...again?)

Hello everyone,

So a while back I was just checking the traffic-stats/graphs from my server on my ISP's site, and I noticed that 2 months ago I had a bandwidth-usage of 300GB, in a couple of days mind you. This was also at a time that I was messing around a bit with the OS's so I didnt really notice it at the time. Today I have FreeBSD 8.2 with IPF ,SSHguard and DirectAdmin installed. And I used to get daily, almost hourly bruteforce attack detected warnings.

Last couple of weeks I got none of those (after upgrading to 8.2 with sshguard). However...today I got one warning of a bruteforce attack on dovecot for 10 failed login attempts. So decided to check the traffic stats again, and turns out I used 17GB of traffic in 1-2 days, at the beginning of this week. This site has only a couple of websites on it, which are all very low traffic.

So do these sudden spikes in traffic mean that my server is hacked, and used for something..like a spambot or something? Cause i'm sure that 17GB wasnt because of me.

Thanks in advance.
 
You'll probably want a network traffic monitoring/analysis tool to find out where the traffic comes from. I use net/wireshark myself because I need it at the university anyway, but it's big and has a bit of a learning curve. There are others.

Fonz
 
Alright, so I installed the mentioned programs, couldnt get wireshark to work properly but got iftop running.

When I check my ISP's bandwidth meter, im still using more bandwidth than I should, is iftop is showing me the following.

myserver => 218.75.48.230

myserver => ip174-100-209-87.adsl2.st

myserver => vrrp.mcast.net

Not a lot of traffic to those at the moment..but still...I have a feeling the top IP-address shouldnt be there.

Now, is this something bad? (probably), and is it possible to get hacked via dovecot or something else than SSH? (since I got most of my bruteforce attacks on dovecot, and I basically got an unguessable password).

Thanks!
 
For security/aide, security/tripwire or security/integrit it might be too late (if you're already hacked)
These need to be installed on secure, unhacked system, because they make database (that you should deep in CD-ROM or DVD-ROM [Your server shouldn't be able to write to them]) from files present on system.
If system was compromised database would contain info about compromised files, and thus it' might bee too lated (image if they patched ssh binary for example, if your database have only info about patched ssh, then you won't be able to detect, that it's patched :) )
 
graudeejs said:
For security/aide, security/tripwire or security/integrit it might be too late (if you're already hacked)
These need to be installed on secure, unhacked system, because they make database (that you should deep in CD-ROM or DVD-ROM [Your server shouldn't be able to write to them]) from files present on system.
If system was compromised database would contain info about compromised files, and thus it' might bee too lated (image if they patched ssh binary for example, if your database have only info about patched ssh, then you won't be able to detect, that it's patched :) )

I see , thanks, do you als know the answer to my last question?:

Linc said:
and is it possible to get hacked via dovecot or something else other than SSH? (since I got most of my bruteforce attacks on dovecot, and I basically have an unguessable password).

Thanks!
 
Everything's possible, especially after Kasperskys statement, that he could hack/crack Intel (If I remember correctly) CPU microcode using JS in browser (or something like that)
 
Linc said:
couldnt get wireshark to work properly
What exactly is the problem? Does it fail to install/build? Does it fail to run? Does it fail to list capturable devices (which is most likely a permissions issue)? Or do you just don't know how to use it, which as I said earlier is understandable because Wireshark does come with a learning curve?

Fonz
 
fonz said:
What exactly is the problem? Does it fail to install/build? Does it fail to run? Does it fail to list capturable devices (which is most likely a permissions issue)? Or do you just don't know how to use it, which as I said earlier is understandable because Wireshark does come with a learning curve?

Fonz

gave a bunch of errors during install and wouldnt run afterwards. But i'm gonna start with a clean OS install this week and make the necessary changes/improvements from the start.
 
So stuff is getting weirder by the day, yesterday (wednesday evening), i wiped the HDD of the server with darik's boot and nuke thats on ultimate boot cd.

So I was expecting for my bandwidth usage to go down to zero after the format, since you can connect tot the server. However, today I checked my ISP's bandwidth charts, and turns out, im still using bandwith...a gig a day it looks like. So how can my server with a wiped HDD that I cant connect to via my domains/ip-address still use bandwidth? Am I missing something here? Or is my ISP playing tricks?

thanks in advance
 
Linc said:
So stuff is getting weirder by the day, yesterday (wednesday evening), i wiped the HDD of the server with darik's boot and nuke thats on ultimate boot cd.

So I was expecting for my bandwidth usage to go down to zero after the format, since you can connect tot the server. However, today I checked my ISP's bandwidth charts, and turns out, im still using bandwith...a gig a day it looks like. So how can my server with a wiped HDD that I cant connect to via my domains/ip-address still use bandwidth? Am I missing something here? Or is my ISP playing tricks?

thanks in advance

meant, since you CAN'T connect.
 
What alse is on the network.
Is the server the only machine in that network?
What kind of firewall are you using in front of that server (if any)

Run the following command on your FreeBSD Machine

Code:
systat -ifstat

this will show you some statistics of your network card

Code:
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average

      Interface           Traffic               Peak                Total
            em0  in      0.081 KB/s          0.130 KB/s            4.882 GB
                 out     0.079 KB/s          0.119 KB/s          171.931 MB

Traffic and Peak is traffic at the moment Total is the total since the interface is active.
Here you can nicely see if your server is busy with network activity.

regards
Johan
 
Sylhouette said:
What alse is on the network.
Is the server the only machine in that network?
What kind of firewall are you using in front of that server (if any)

Run the following command on your FreeBSD Machine

Code:
systat -ifstat

this will show you some statistics of your network card

Code:
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average

      Interface           Traffic               Peak                Total
            em0  in      0.081 KB/s          0.130 KB/s            4.882 GB
                 out     0.079 KB/s          0.119 KB/s          171.931 MB

Traffic and Peak is traffic at the moment Total is the total since the interface is active.
Here you can nicely see if your server is busy with network activity.

regards
Johan

Hi Johan, I wiped the server's HDD clean, so there is no OS installed, but im still using bandwidth somehow.
The server is in a rack with other dedicated servers (from the same ISP).

Obviously I cant even connect to the server via the IP/ssh or a domain.
 
Silly question, but you said your server is in the same rack with other dedicated servers... could your ISP be sharing your IP with another server (doing NAT)? If you have a dedicated public ip for your server, could it be possible that one of those other servers is incorrectly configured and using your ip?

Another issue could be the bandwidth graph it self, there have been times where (I fallen to this) a graph is configured to monitor the wrong ip accidentally. Have you talked to your ISP where your dedicated server is hosted? If your current server has no OS, and therefore not usable for anything, then most likely its another system connected on the same network segment as you.

Just a thought
 
Do you need all these services exposed to the Internet?

Code:
$ nmap -Av 218.75.48.230

Starting Nmap 5.59BETA1 ( http://nmap.org ) at 2011-09-09 12:27 EST
NSE: Loaded 63 scripts for scanning.
NSE: Script Pre-scanning.
Initiating Ping Scan at 12:27
Scanning 218.75.48.230 [2 ports]
Completed Ping Scan at 12:27, 0.34s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 12:27
Completed Parallel DNS resolution of 1 host. at 12:27, 1.19s elapsed
Initiating Connect Scan at 12:27
Scanning 218.75.48.230 [1000 ports]
Discovered open port 111/tcp on 218.75.48.230
Discovered open port 8080/tcp on 218.75.48.230
Discovered open port 199/tcp on 218.75.48.230
Discovered open port 113/tcp on 218.75.48.230
Discovered open port 21/tcp on 218.75.48.230
Discovered open port 22/tcp on 218.75.48.230
Discovered open port 2100/tcp on 218.75.48.230
Discovered open port 32784/tcp on 218.75.48.230
Discovered open port 1521/tcp on 218.75.48.230
Discovered open port 9090/tcp on 218.75.48.230
Discovered open port 32785/tcp on 218.75.48.230
Discovered open port 9080/tcp on 218.75.48.230
Connect Scan Timing: About 42.97% done; ETC: 12:29 (0:00:41 remaining)
Discovered open port 32768/tcp on 218.75.48.230
Discovered open port 1311/tcp on 218.75.48.230
Discovered open port 5001/tcp on 218.75.48.230
Discovered open port 2809/tcp on 218.75.48.230
Completed Connect Scan at 12:28, 71.23s elapsed (1000 total ports)
Initiating Service scan at 12:28
Scanning 16 services on 218.75.48.230
Completed Service scan at 12:30, 75.10s elapsed (16 services on 1 host)
Initiating RPCGrind Scan against 218.75.48.230 at 12:30
Completed RPCGrind Scan against 218.75.48.230 at 12:30, 4.10s elapsed (2 ports)
NSE: Script scanning 218.75.48.230.
Initiating NSE at 12:30
Completed NSE at 12:30, 14.58s elapsed
Nmap scan report for 218.75.48.230
Host is up (0.74s latency).
Not shown: 983 closed ports

Code:
$ snmpwalk -c public -v 1 218.75.48.230
SNMPv2-MIB::sysDescr.0 = STRING: Linux dqcz.localdomain 2.4.21-37.ELsmp #1 SMP Wed Sep 7 13:28:55 EDT 2005 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (3090454252) 357 days, 16:35:42.52
SNMPv2-MIB::sysContact.0 = STRING: Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
SNMPv2-MIB::sysName.0 = STRING: dqcz.localdomain
SNMPv2-MIB::sysLocation.0 = STRING: Unknown (edit /etc/snmp/snmpd.conf)
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01
 
Code:
$ snmpwalk -c public -v 1 218.75.48.230
SNMPv2-MIB::sysDescr.0 = STRING: Linux dqcz.localdomain 2.4.21-37.ELsmp #1 SMP Wed Sep 7 13:28:55 EDT 2005 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (3090454252) 357 days, 16:35:42.52
SNMPv2-MIB::sysContact.0 = STRING: Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
SNMPv2-MIB::sysName.0 = STRING: dqcz.localdomain
SNMPv2-MIB::sysLocation.0 = STRING: Unknown (edit /etc/snmp/snmpd.conf)
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01

Ok, I would take a guess that you do have another system with the same public ip address configured. The above system is a RedHat Enterprise linux system with hostname 'dqcz', and by the kernel version I would guess its a RHEL 3 version. Also, its uptime is 357 days, so it more than likely its not the same server you are working with.
 
redw0lfx said:
Silly question, but you said your server is in the same rack with other dedicated servers... could your ISP be sharing your IP with another server (doing NAT)? If you have a dedicated public ip for your server, could it be possible that one of those other servers is incorrectly configured and using your ip?

Another issue could be the bandwidth graph it self, there have been times where (I fallen to this) a graph is configured to monitor the wrong ip accidentally. Have you talked to your ISP where your dedicated server is hosted? If your current server has no OS, and therefore not usable for anything, then most likely its another system connected on the same network segment as you.

Just a thought

Yes, thats what comes to mind now, my traffic/bandwidth is now +4GB since the day I wiped my HDD (2 days ago). I'm gonna contact my ISP this weekend to ask if they can verify the bandwidth monitor.
edit: on your first thought, possible...but very unlikely.
 
shitson said:
Do you need all these services exposed to the Internet?

Code:
$ nmap -Av 218.75.48.230

Starting Nmap 5.59BETA1 ( http://nmap.org ) at 2011-09-09 12:27 EST
NSE: Loaded 63 scripts for scanning.
NSE: Script Pre-scanning.
Initiating Ping Scan at 12:27
Scanning 218.75.48.230 [2 ports]
Completed Ping Scan at 12:27, 0.34s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 12:27
Completed Parallel DNS resolution of 1 host. at 12:27, 1.19s elapsed
Initiating Connect Scan at 12:27
Scanning 218.75.48.230 [1000 ports]
Discovered open port 111/tcp on 218.75.48.230
Discovered open port 8080/tcp on 218.75.48.230
Discovered open port 199/tcp on 218.75.48.230
Discovered open port 113/tcp on 218.75.48.230
Discovered open port 21/tcp on 218.75.48.230
Discovered open port 22/tcp on 218.75.48.230
Discovered open port 2100/tcp on 218.75.48.230
Discovered open port 32784/tcp on 218.75.48.230
Discovered open port 1521/tcp on 218.75.48.230
Discovered open port 9090/tcp on 218.75.48.230
Discovered open port 32785/tcp on 218.75.48.230
Discovered open port 9080/tcp on 218.75.48.230
Connect Scan Timing: About 42.97% done; ETC: 12:29 (0:00:41 remaining)
Discovered open port 32768/tcp on 218.75.48.230
Discovered open port 1311/tcp on 218.75.48.230
Discovered open port 5001/tcp on 218.75.48.230
Discovered open port 2809/tcp on 218.75.48.230
Completed Connect Scan at 12:28, 71.23s elapsed (1000 total ports)
Initiating Service scan at 12:28
Scanning 16 services on 218.75.48.230
Completed Service scan at 12:30, 75.10s elapsed (16 services on 1 host)
Initiating RPCGrind Scan against 218.75.48.230 at 12:30
Completed RPCGrind Scan against 218.75.48.230 at 12:30, 4.10s elapsed (2 ports)
NSE: Script scanning 218.75.48.230.
Initiating NSE at 12:30
Completed NSE at 12:30, 14.58s elapsed
Nmap scan report for 218.75.48.230
Host is up (0.74s latency).
Not shown: 983 closed ports

Code:
$ snmpwalk -c public -v 1 218.75.48.230
SNMPv2-MIB::sysDescr.0 = STRING: Linux dqcz.localdomain 2.4.21-37.ELsmp #1 SMP Wed Sep 7 13:28:55 EDT 2005 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (3090454252) 357 days, 16:35:42.52
SNMPv2-MIB::sysContact.0 = STRING: Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
SNMPv2-MIB::sysName.0 = STRING: dqcz.localdomain
SNMPv2-MIB::sysLocation.0 = STRING: Unknown (edit /etc/snmp/snmpd.conf)
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01

redw0lfx said:
Code:
$ snmpwalk -c public -v 1 218.75.48.230
SNMPv2-MIB::sysDescr.0 = STRING: Linux dqcz.localdomain 2.4.21-37.ELsmp #1 SMP Wed Sep 7 13:28:55 EDT 2005 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (3090454252) 357 days, 16:35:42.52
SNMPv2-MIB::sysContact.0 = STRING: Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
SNMPv2-MIB::sysName.0 = STRING: dqcz.localdomain
SNMPv2-MIB::sysLocation.0 = STRING: Unknown (edit /etc/snmp/snmpd.conf)
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01

Ok, I would take a guess that you do have another system with the same public ip address configured. The above system is a RedHat Enterprise linux system with hostname 'dqcz', and by the kernel version I would guess its a RHEL 3 version. Also, its uptime is 357 days, so it more than likely its not the same server you are working with.

guys, that not my server, on the 'iftop text' I posted was my servers's IP on the left (replaced by 'myserver') and the traffic destination on the right. So the IP you listed is probably of someone stealing traffic. Atleast..when I google that IP, it shows some hacker-blacklists sites with the ip on it.
 
Linc said:
guys, that not my server, on the 'iftop text' I posted was my servers's IP on the left (replaced by 'myserver') and the traffic destination on the right. So the IP you listed is probably of someone stealing traffic. Atleast..when I google that IP, it shows some hacker-blacklists sites with the ip on it.

:e

Do you have a list of services you run? - Do you have a firewall on your box?
 
shitson said:
:e

Do you have a list of services you run? - Do you have a firewall on your box?

Since the HDD is wiped, I cant give you the list, since I dont have them in my mind.Yes had IP filter.

Anyway, contacted my ISP, they reported back that the bandwidth meter was not correct, and theyre gonna do maintenance on the switches tonight. Hopefully this will solve the problem... )
 
Back
Top