Hello.
I have a security issue. Today I logged at my test machine and I discovered that it is a bit laggy. I checked to see what is going on and I saw a root process named bsd eating all cpu. I began some simple steps to see what is going on. I dont know how to debug such kind of problems. To me it seems that my machine is hacked. This machine is not very important for our company - it is a test copy of our original www server, but our others machines are very similar.
Here are details that I collected.
top:
sockstat:
netstat:
I tried to find the process involved:
parts of message log:
There are some strange entries in userslog:
I am the only one knowing the root password, and it is not me that has beed adding or removing users on this date.
My freebsd version:
I am attaching a file (hacked.zip) containg some traffic captured with tcpdump for 64.85.170.145 and the executable involved.
I just killed all processes named bsd with killall -9 bsd, but I am afraid that this may happen again on others important machines in my network (also with bsd 8.1). Need advices please! How to understand from where this process has been run? What can I do further. I'll give additional details if required.
10x in advance.
I have a security issue. Today I logged at my test machine and I discovered that it is a bit laggy. I checked to see what is going on and I saw a root process named bsd eating all cpu. I began some simple steps to see what is going on. I dont know how to debug such kind of problems. To me it seems that my machine is hacked. This machine is not very important for our company - it is a test copy of our original www server, but our others machines are very similar.
Here are details that I collected.
top:
Code:
last pid: 32745; load averages: 1.22, 1.23, 1.16 up 0+22:19:21 10:37:27
77 processes: 3 running, 74 sleeping
CPU: 0.4% user, 0.0% nice, 99.6% system, 0.0% interrupt, 0.0% idle
Mem: 100M Active, 322M Inact, 144M Wired, 111M Buf, 430M Free
Swap: 2015M Total, 2015M Free
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
32313 root 1 117 0 3288K 1212K RUN 84:05 97.17% bsd
...
Code:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
...
root bsd 32313 0 tcp4 89.25.83.163:40015 64.85.170.131:45295
root bsd 32313 1 tcp4 89.25.83.163:40015 64.85.170.131:45295
root bsd 32313 2 tcp4 89.25.83.163:40015 64.85.170.131:45295
root bsd 32313 5 tcp4 89.25.83.163:40015 64.85.170.131:45295
root bsd 32313 6 tcp4 89.25.83.163:61380 64.85.170.145:40808
root bsd 32313 7 tcp4 89.25.83.163:33833 64.85.170.145:40808
root bsd 32313 8 tcp4 89.25.83.163:47559 64.85.170.145:40808
root bsd 32313 9 tcp4 89.25.83.163:52354 64.85.170.145:40808
root bsd 32313 10 tcp4 89.25.83.163:62054 64.85.170.145:40808
root bsd 32313 11 tcp4 89.25.83.163:32914 64.85.170.145:40808
root bsd 32313 12 tcp4 89.25.83.163:26434 64.85.170.145:40808
......about 16000 rows
Code:
#:> sockstat | grep 64.85.170.145 | wc -l
16376
Code:
#:> netstat -n | grep 64.85.170.145
tcp4 121 0 89.25.83.163.23093 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.61761 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.12579 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.37957 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.12753 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.22634 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.47220 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.20992 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.55763 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.27006 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.52767 64.85.170.145.40808 CLOSED
tcp4 0 0 89.25.83.163.33022 64.85.170.145.40808 CLOSED
...... about 8200 rows, last one is opened
tcp4 0 0 89.25.83.163.37444 64.85.170.145.40808 ESTABLISHED
Code:
#:> netstat -n | grep 64.85.170.145 | grep 64.85.170.145 | wc -l
8188
I tried to find the process involved:
Code:
#:> find / -type file -name bsd
/bsd
Code:
#:> ls /bsd
-rwxrwxr-x 1 root wheel 23241 Dec 4 08:20 /bsd
parts of message log:
Code:
Dec 4 12:18:23 test proftpd[14265]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:23 test proftpd[14266]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec 4 12:18:23 test proftpd[14267]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:23 test proftpd[14269]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:24 test proftpd[14270]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:24 test proftpd[14268]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec 4 12:18:24 test proftpd[14271]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec 4 12:18:27 test proftpd[14272]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:27 test proftpd[14277]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:27 test proftpd[14278]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:27 test proftpd[14279]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - client sent too-long command, ignoring
Dec 4 12:18:36 test proftpd[14282]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
Dec 4 12:18:36 test proftpd[14281]: 192.168.0.3 (80.245.33.225[80.245.33.225]) - ProFTPD terminating (signal 11)
...
Dec 4 12:46:13 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec 4 12:46:46 test last message repeated 10 times
Dec 4 12:48:43 test last message repeated 35 times
Dec 4 12:51:38 test last message repeated 47 times
Dec 4 12:51:40 test kernel: kern.maxfiles limit exceeded by uid 26, please see tuning(7).
Dec 4 12:51:43 test apcupsd[942]: Communications with UPS restored.
Dec 4 12:51:43 test syslogd: /dev/console: Too many open files in system: Too many open files in system
Dec 4 12:51:43 test apcupsd[942]: apcserver: accept error. ERR=Too many open files in system
Dec 4 12:51:43 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec 4 12:51:50 test last message repeated 2 times
Dec 4 12:52:34 test last message repeated 3 times
Dec 4 12:52:44 test apcupsd[942]: Communications with UPS lost.
Dec 4 12:52:44 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
Dec 4 12:52:49 test kernel: kern.maxfiles limit exceeded by uid 0, please see tuning(7).
...
...
Dec 4 12:55:41 test kernel: pid 50506 (httpd), uid 80: exited on signal 11
Dec 4 12:55:41 test kernel: pid 50507 (httpd), uid 80: exited on signal 11
Dec 4 12:55:41 test kernel: pid 50508 (httpd), uid 80: exited on signal 11
Dec 4 12:55:41 test kernel: pid 50509 (httpd), uid 80: exited on signal 11
Dec 4 12:55:41 test kernel: pid 50510 (httpd), uid 80: exited on signal 11
Dec 4 12:55:43 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Dec 4 12:55:43 test kernel: pid 50513 (httpd), uid 80: exited on signal 11
Dec 4 12:55:44 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Dec 4 12:55:44 test kernel: pid 50514 (httpd), uid 80: exited on signal 11
Dec 4 12:55:44 test kernel: pid 50515 (httpd), uid 80: exited on signal 11
Dec 4 12:55:45 test kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
...
There are some strange entries in userslog:
Code:
2010-12-02 04:04:59 [unknown] u232004(0):daemon(1):Administrator Manager:/var/tmp:/bin/sh
2010-12-02 06:28:18 [unknown] u232004(0) account removed
My freebsd version:
Code:
#:> uname -a
FreeBSD test.pulsar.bg 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Tue Aug 10 16:01:50 EEST 2010
root@test.pulsar.bg:/usr/obj/usr/src/sys/TESTPC i386
I am attaching a file (hacked.zip) containg some traffic captured with tcpdump for 64.85.170.145 and the executable involved.
I just killed all processes named bsd with killall -9 bsd, but I am afraid that this may happen again on others important machines in my network (also with bsd 8.1). Need advices please! How to understand from where this process has been run? What can I do further. I'll give additional details if required.
10x in advance.