Help. My FreeBSD 8.1R machine may be hacked!!?

edhunter

edhunter

Hello.
I have a security issue. Today I logged at my test machine and I discovered that it is a bit laggy. I checked to see what is going on and I saw a root process named bsd eating all cpu. I began some simple steps to see what is going on. I dont know how to debug such kind of problems. To me it seems that my machine is hacked. This machine is not very important for our company - it is a test copy of our original www server, but our others machines are very similar.
Here are details that I collected.
top:
Code: 
last pid: 32745;  load averages:  1.22,  1.23,  1.16    up 0+22:19:21  10:37:27
77 processes:  3 running, 74 sleeping
CPU:  0.4% user,  0.0% nice, 99.6% system,  0.0% interrupt,  0.0% idle
Mem: 100M Active, 322M Inact, 144M Wired, 111M Buf, 430M Free
Swap: 2015M Total, 2015M Free

  PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
32313 root         1 117    0  3288K  1212K RUN     84:05 97.17% bsd
...
sockstat:
Code: 
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
...
root     bsd        32313 0  tcp4   89.25.83.163:40015    64.85.170.131:45295
root     bsd        32313 1  tcp4   89.25.83.163:40015    64.85.170.131:45295
root     bsd        32313 2  tcp4   89.25.83.163:40015    64.85.170.131:45295
root     bsd        32313 5  tcp4   89.25.83.163:40015    64.85.170.131:45295
root     bsd        32313 6  tcp4   89.25.83.163:61380    64.85.170.145:40808
root     bsd        32313 7  tcp4   89.25.83.163:33833    64.85.170.145:40808
root     bsd        32313 8  tcp4   89.25.83.163:47559    64.85.170.145:40808
root     bsd        32313 9  tcp4   89.25.83.163:52354    64.85.170.145:40808
root     bsd        32313 10 tcp4   89.25.83.163:62054    64.85.170.145:40808
root     bsd        32313 11 tcp4   89.25.83.163:32914    64.85.170.145:40808
root     bsd        32313 12 tcp4   89.25.83.163:26434    64.85.170.145:40808
......about 16000 rows
Code: 
#:> sockstat | grep 64.85.170.145 | wc -l
   16376
netstat:
Code: 
#:> netstat -n | grep 64.85.170.145
tcp4     121      0 89.25.83.163.23093     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.61761     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.12579     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.37957     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.12753     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.22634     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.47220     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.20992     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.55763     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.27006     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.52767     64.85.170.145.40808    CLOSED
tcp4       0      0 89.25.83.163.33022     64.85.170.145.40808    CLOSED
...... about 8200 rows, last one is opened
tcp4       0      0 89.25.83.163.37444     64.85.170.145.40808    ESTABLISHED
Code: 
#:> netstat -n | grep 64.85.170.145 | grep 64.85.170.145 | wc -l
    8188

I tried to find the process involved:
Code: 
#:> find / -type file -name bsd
/bsd
Code: 
#:> ls /bsd
-rwxrwxr-x  1 root  wheel  23241 Dec  4 08:20 /bsd

parts of message log:
Code: 
Dec  4 12:18:23 test proftpd[14265]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:23 test proftpd[14266]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec  4 12:18:23 test proftpd[14267]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:23 test proftpd[14269]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:24 test proftpd[14270]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:24 test proftpd[14268]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec  4 12:18:24 test proftpd[14271]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec  4 12:18:27 test proftpd[14272]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:27 test proftpd[14277]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:27 test proftpd[14278]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:27 test proftpd[14279]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec  4 12:18:36 test proftpd[14282]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec  4 12:18:36 test proftpd[14281]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)

...
Dec  4 12:46:13 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec  4 12:46:46 test last message repeated 10 times
Dec  4 12:48:43 test last message repeated 35 times
Dec  4 12:51:38 test last message repeated 47 times
Dec  4 12:51:40 test kernel: kern.maxfiles limit exceeded by uid 26, please see tuning(7).
Dec  4 12:51:43 test apcupsd[942]: Communications with UPS restored.
Dec  4 12:51:43 test syslogd: /dev/console: Too many open files in system: Too many open files in system
Dec  4 12:51:43 test apcupsd[942]: apcserver: accept error. ERR=Too many open files in system
Dec  4 12:51:43 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec  4 12:51:50 test last message repeated 2 times
Dec  4 12:52:34 test last message repeated 3 times
Dec  4 12:52:44 test apcupsd[942]: Communications with UPS lost.
Dec  4 12:52:44 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec  4 12:52:49 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
...
...
Dec  4 12:55:41 test kernel: pid 50506 (httpd), uid 80: exited on signal 11
Dec  4 12:55:41 test kernel: pid 50507 (httpd), uid 80: exited on signal 11
Dec  4 12:55:41 test kernel: pid 50508 (httpd), uid 80: exited on signal 11
Dec  4 12:55:41 test kernel: pid 50509 (httpd), uid 80: exited on signal 11
Dec  4 12:55:41 test kernel: pid 50510 (httpd), uid 80: exited on signal 11
Dec  4 12:55:43 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Dec  4 12:55:43 test kernel: pid 50513 (httpd), uid 80: exited on signal 11
Dec  4 12:55:44 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Dec  4 12:55:44 test kernel: pid 50514 (httpd), uid 80: exited on signal 11
Dec  4 12:55:44 test kernel: pid 50515 (httpd), uid 80: exited on signal 11
Dec  4 12:55:45 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
...

There are some strange entries in userslog:
Code: 
2010-12-02 04:04:59 [unknown] u232004(0):daemon(1):Administrator Manager:/var/tmp:/bin/sh
2010-12-02 06:28:18 [unknown] u232004(0) account removed
I am the only one knowing the root password, and it is not me that has beed adding or removing users on this date.


My freebsd version:
Code: 
#:> uname -a
FreeBSD test.pulsar.bg 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Tue Aug 10 16:01:50 EEST 2010
  root@test.pulsar.bg:/usr/obj/usr/src/sys/TESTPC  i386

I am attaching a file (hacked.zip) containg some traffic captured with tcpdump for 64.85.170.145 and the executable involved.

I just killed all processes named bsd with killall -9 bsd, but I am afraid that this may happen again on others important machines in my network (also with bsd 8.1). Need advices please! How to understand from where this process has been run? What can I do further. I'll give additional details if required.
10x in advance.
 
Just take the machine offline, wipe and reinstall. Don't forget to update it and harden it properly before connecting it to the internet again.

You really shouldn't treat test or production boxes differently when they're connected to the internet.

Edit: You may have been pwned by this: [thread=19849]Backdoor discovered in ProFTPd[/thread]
 
Thank you SirDice :)
I had some suspicions about proftp, bacause previously I had some issues with high cpu usage and many connections, and on 1.dec I reinstalled it. Now I will delete all proftpd files and sources and will reinstall it again.
10x
 
It's better to wipe the whole machine and start over. Because they got in there's no telling what else has been changed. They may have installed a rootkit that guarantees their access and keeps everything hidden.

When stuff like this happens don't take chances, you simply cannot trust anything on that machine anymore. Wipe and start over.
 
edhunter said:
Now I will delete all proftpd files and sources
Click to expand...
Did you just downloaded sources from proftpd site ?

I believe this cannot be happed when you install from ports (MD5 sign), isnt?
 
Alt said:
I believe this cannot be happed when you install from ports (MD5 sign), isnt?
Click to expand...
There was a small window of time where the port would install the backdoored version.
 
I installed it from Ports (make install clean).
I cant wipe everything. Our web developer has some unfinished work in progress in /usr/local/www/.... and in /var/db/mysql/...

I checked the system for modified files in the last couple of days (using find / -mtime -5d). It seems that the important part of web work in progress is unmodified. I shortened the list of suspected files to:
Code: 
/bsd			root	wheel	-rwxrwxr-x
/etc/group		root	wheel	-rw-r--r--
/etc/master.passwd	root	wheel	-rw-------
/etc/passwd		root	wheel	-rw-r--r--
/etc/pwd.db		root	wheel	-rw-r--r--
/etc/rc.local		root	wheel	-r----x--t
/etc/spwd.db		root	wheel	-rw-------
/usr/include/gpm2.h	root	wheel	-rwxrwxrwx

/etc/bsd is the "virus",
/usr/include/gpm2.h contains my freebsd (non root) user:password in plain text, and may be this file was transmitted to foreign address 64.85.170.131
/etc/rc.local is empty - I thing it may has been used to run /bsd on boot, because the /bsd file is dated 4.dec, but the machine was rebooted yesterday, and yet /bsd process active today

The system must be up and running for a few more days, I can not reinstall it now. So I will change all my passwords for now. Of course proftpd will be reinstalled too or temporary I will use FreeBSD own ftpd daemon.
 
If you got rootkit its nearly impossible to clean it, so reinstalling is only good way..
But i think its nothing bad if you copy your mysql data etc
 
Yes, backup the data. Do wipe the machine. If there is a rootkit on there commands like find, netstat, sockstat etc. may have been altered. You simply cannot trust any of the executables on that box anymore.

Oh, if possible, could you post that bsd file? I'd like to take it apart and see what it does :e
 
You don't know that. Looking at your cpu usage, it could be a part of botnet doing hash attacks or similar. You should inspect the program with truss or gdb to find out what it does. In any case, a lot of malware encrypts communication these days. We used to find a stockpile of stolen info from various computers, just by analyzing communication of shitware programs that uploaded their collected data to FTP servers, and did it plaintext. Perhaps that's why you don't see "anything useful" there.

Up The Irons.
 
Found it. I'll take it apart when I get home. Time to fire up good old IDA pro :stud
 
I have taken the binary out. Don't want it proliferating from here, or people shooting themselves in the foot by running it ;)
 
Hello,

I have found this thread via Google, and have had the exact same thing happen to me. Pretty much identical to edhunter. I, however, am not a very good admin, and have personal websites running on the box. Is there anyone, anywhere, I can hire to help me out with this problem?

Thanks for any help
 
It appears I can no longer even establish a connection via ssh. I have 2 connections, but I am trying to change passwords, and opening new ssh connections but it just closes the putty window on login.

Anything I can do to hold this off until I can move to another server? Something? anything? I really need help, but can't allow my sites to be down for too long.
 
My guess is that they converted your server into a spam spewing zombie. The massive amounts of data probably saturates your internet connection.

Take it offline asap. Call the hosting company and have them pull the plug.
 
FTP is now off. I can now access through the window I still have open. Is there any way to block any ssh from anything other then my current IP? I have IPFW.

Any other tips that can help would be MUCH appreciated - I understand I need to wipe the server, but I have to hold off until I can purchase another one and get it up and running.

So anything I can do would be appreciated. I still have one terminal open.
 
I understand your reluctance but there's really nothing you can do. Even if you close off ssh there's no way to tell if they haven't installed some other way into your server.

Just take it offline and start writing apology letters to your customers.
 
I have notified the company to wipe the machine. This only has personal websites so I have no clients. So is it confirmed this exploit was done by ProFTP? Is it still a compromised port? I just installed it last week (from ports) - not even sure on the version.
 
since the binary is removed can only guess what it was doing.

it was very likely to at least be scanning for new victims.

you welcome to send me a link for the binary in PM and I will check into it further.

proftpd 1.3.3c has this vuln patched.
 
You must log in or register to reply here.
Back
Top