Recent botnet installed as root

E

ernie

I just had 3 machines 6.2-STABLE, 6.3-STABLE, 7.2-RELEASE, all infected on the same day with a botnet. In the /tmp dir there were two abnormal files
Code: 
-rw-r--r--   1 root   wheel     0B Mar 22 15:58 null
-r-xr-xr-x   1 root   wheel   393K Mar 22 07:05 talkngbsd

MD5 (talkngbsd) = 5b2744053752a93462d6fec7c2347c24

There were multiple instances of talkngbsd running, and it was IP spoofing and attacking an overseas server on port 80, no doubt trying to do a DOS. I have no idea how the program got installed as root, there are no user accounts on any of the machines, no ssh enabled, the only thing they have in common is Apache and sendmail running. Are there any know exploits for those that could escalate privileges?
 
What version of www/apache port? All of the FreeBSD versions that you mention have been end of life for a while meaning there are no security updates for them even if serious security holes are found. Clean up the mess and upgrade to at least 7.4 or preferably 8.2.
 
/var/db/pkg/apache-2.2.4_2 on two of them but I just realized the third machine didn't have Apache installed, so that kind of rules out apache.

From rc.conf the processes that were common to all 3

Code: 
bsnmpd_enable="YES"
gateway_enable="YES"
inetd_enable="YES"
ntpdate_enable="YES"
sendmail_enable="YES"
usbd_enable="YES"
 
xibo said:
what is enabled via inetd?
Click to expand...

Code: 
#
#ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l
#telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd
login   stream  tcp     nowait  root    /usr/libexec/rlogind    rlogind

Telnet and ftp are usually disabled and activated as needed, I normally login with rlogin, the su password is quite complex.
 
ernie said:
#
#ftp stream tcp nowait root /usr/libexec/ftpd ftpd -l
#telnet stream tcp nowait root /usr/libexec/telnetd telnetd
login stream tcp nowait root /usr/libexec/rlogind rlogind


Telnet and ftp are usually disabled and activated as needed, I normally login with rlogin, the su password is quite complex.
Click to expand...

I would suggest using OpenSSH instead of rlogin. Also, you are running Apache 2.2.4 which has some security vulnerabilities. Current version of Apache is 2.2.22 (unless you have kept it patched). If they wrote to /tmp, I wouldn't doubt they came in through apache specially if mod_php is enabled. However, rlogin could have unknown exploits as well.

I would look at the apache logs to see if any weird urls were called, mainly stuff with \x0a\x78\x... etc. I would also check history on the root user and perhaps your authentication logs. However, keep in mind that they could have cleaned up after themselves.

ernie said:
I have no idea how the program got installed as root, there are no user accounts on any of the machines, no ssh enabled, the only thing they have in common is Apache and sendmail running. Are there any know exploits for those that could escalate privileges?
Click to expand...

You have to look at what services are exposed to the internet, and determine if there are any known exploits for them that would give at least a shell prompt. Once you have a shell prompt, all bets are off, as you can then attempt to exploit any installed binary to escalate privileges, for example, through mutt, vi, etc.

Once you are satisfied no more information can be obtained from the compromised system, you should re-install your FreeBSD system with a current version, as who knows what else they could have done.
 
I think it might have been telnetd, I hadn't noticed the security advisory from Dec 2011, which is odd as I am on the mail list. I use rlogind only internally, ssh fills the logs with probe attempts. Upgrading the 7.2-RELEASE machine to 7.4-RELEASE will be easy, the 6.2-STABLE, 6.3-STABLE machines I am not quite sure how to go about. Welcome any suggestions.
 
Having your logs fill with ssh probe attempts is a lesser evil than having your boxes rooted. It's also easily avoided by running ssh on a non-standard port.

Not using plain text protocols like rlogin to log into your systems is the absolute number one security measure that most system admins should follow. That's been pretty standard for many years.
 
Ah, yes, if telnetd is exposed to the internet it is also a possiblity. As for upgrading the 7.2 system to 7.4, do you really trust that you found all rootkits/botnets installed on the system and removed them, as well as blocked any backdoors they might have setup?
 
redw0lfx said:
Ah, yes, if telnetd is exposed to the internet it is also a possiblity. As for upgrading the 7.2 system to 7.4, do you really trust that you found all rootkits/botnets installed on the system and removed them, as well as blocked any backdoors they might have setup?
Click to expand...

I find that when you do an upgrade, any hacked binaries get overwritten, mergemaster should point out any diffs. I discovered that telnetd had been deleted by the hacker, on the 3 machines concerned, I can't think why, perhaps the bot binary intercepts port 23? I will run it up on an isolated test network and see what ports it opens. Two of the 3 machines were hacked within seonds of each other, I suspect it was automated.
 
jem said:
Having your logs fill with ssh probe attempts is a lesser evil than having your boxes rooted.
Click to expand...
Just use something like security/sshguard.

Not using plain text protocols like rlogin to log into your systems is the absolute number one security measure that most system admins should follow. That's been pretty standard for many years.
Click to expand...
I fully agree. Don't use telnet or rlogin. That's asking for trouble. Even if you only use it 'internally'.

It quite possible they came in through a web application, started sniffing and grabbed the root password you typed in on an insecure connection. Once root was obtained they had full reign over your servers.

Backup your data and wipe everything. Reinstall using a current and supported version.
 
ernie said:
I discovered that telnetd had been deleted by the hacker, on the 3 machines concerned, I can't think why, perhaps the bot binary intercepts port 23?
Click to expand...
No, it's to prevent others from getting in through the same hole. They don't like it when some other hacker takes over their pwn3d servers.
 
ernie said:
Telnet and ftp are usually disabled and activated as needed, I normally login with rlogin, the su password is quite complex.
Click to expand...
Complexity of the password is irrelevant as it travels the wire in the clear. Anybody with a sniffer can grab it. As noted by others use ssh(1).
 
attached are some 'logs' from my server

Code: 
MD5 (talkngbsd) = 5b2744053752a93462d6fec7c2347c24

MD5 (dnsbang4b) = 33787e6319263a2cee2b5c1fdc11c63d

MD5 (dnskill4bsd) = 72a54adf0ce791b52781aad95c20333c
  
MD5 (ro.bang) = fe291b5bbe47b10b14ca3193a83a2d1e
 
Your description of the attack matches my case as well. I haven't yet found another report/post match, so I wanted to share my findings/facts….

At about 2012-03-22 17:45:45 UTC, my server started transmitting ~15Mbps of spoofed UDP/DNS traffic.
UDP any :>1024 --> any:53

I have some NetFlow data, just a small capture… I didn't do much analysis on the destination (anyIP:53)… but it appeared random… not targeted at any of the 13 ish whatever root servers or anything, not that they might not have been mixed in (small sample below). The hack caused a DoS to me, as my ISP uplink is just 10Mbps, but the edge router is running RPF + has ACL's and the spoofed traffic never hit the net.

The server was a small box running FreeBSD 7.2… installed in Sept 2009. Other than taking on a role in my DNS architecture, after another server died, the box wasn't doing too much and being that I was only running SSH, NTP and DNS (BIND 9.4.3-P2)… I (obviously wrongly, lesson learned), assumed the box was not much of an attack vector or target… I wasn't patching/updating it. My GUESS is the attach was against BIND - I still need to do some research, but probably some buffer overflow exploit is my guess. I'm not all that nix/server savvy, so not all sure where to start with that analysis actually.

I suspect the attack could have been via the TCP side of DNS…. but that is pure speculation… only as I haven't done anything on the server, but it hasn't reoccurred (yet). I did put some ACL's around the box so that it can only initiate TCP connections (and they are logged - the box isn't doing anything suspicious there), and NTP and DNS is allowed in from the net (and all ICMP)…. so the box is sheltered from the net of any TCP SYN attempts, etc. The ACL's have isolated the box, and I was hoping to do some analysis, but then the box had a "reboot" command issued (I assume built into the script/binary that was loaded)… and after reboot the binaries remain in the tmp folder, but none of processes are running. The reboot happened at Fri Mar 23 11:47. UTC.


** TIA --- any comments / suggestions / help if very much welcome….


Fact summary:
Infected/hacked at 2012-03-22 17:44UTC
FreeBSD 7.2 release
services running, SSH, telnet (firewalled from net access - only allowed locally, not used - emergency only), BIND (BIND 9.4.3-P2) and NTP
~ 15Mbps of 'random' spoofed UDP traffic to dest port 53 (DNS)
binaries/scripts were running as root -- not sure how (BIND doesn't run as root, root is not allowed remote login, etc)
TCP established connections to 212.112.241.58:81 (I didn't have a method to capture/sniff the traffic)
At about Mar 23 05:27:54.989 UTC I finished isolating the box (ACL's around the box, to allow future analysis on the attack) - this broke the above TCP connections.
Server did a "reboot" on Fri Mar 23 11:47. UTC
Box left as-is since, haven't had time to debug.

I just noticed that sendmail was running… and not sure what TCP 3130 is off-hand. TCP 25 is isolated to only localhost/127.0.0.1. Didn't show up in nmap scans. This is new to me. Possibly the attack point??
 
kleinart said:
binaries/scripts were running as root -- not sure how (BIND doesn't run as root, root is not allowed remote login, etc)
Click to expand...
One possible scenario is that they brute-forced your secure shell. Once inside they used a local exploit (or perhaps the telnetd bug) to gain root. This is why you should never take local bugs lightly.
 
Hi,
I recently found this on a server:
Code: 
# md5 talkngbsd
md5 talkngbsd
MD5 (talkngbsd) = 0d41ba8126893455fc9372e15e01dfcb
Code: 
# ls -l
-rwxr-xr-x  1 root  wheel  401972 Sep 16 11:28 talkngbsd
Code: 
# file talkngbsd
talkngbsd: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 5.0 (500043), statically linked, FreeBSD-style, stripped
Strings: http://pastie.org/4790447
I don't know / can't figure out which vector the attacker used, everythings just outdated, apache with php (also webmin..) is running, so I expect the usual suspects, but who knows...
Code: 
7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 19:59:52 UTC 2008     root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

There is a dropbear running:
Code: 
root     dropbear   637   4  tcp4   *:3129                *:*
which is not installed.
Code: 
# ps xawwj | grep 637
root     637     1   637   637    0 Is    ??    0:00.01 csh (csh)  (dropbear)
Code: 
# fstat -p 637
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     dropbear     637 root /             2 drwxr-xr-x     512  r
root     dropbear     637   wd /             2 drwxr-xr-x     512  r
root     dropbear     637 text /usr     236554 -rwxr-xr-x  570988  r
root     dropbear     637    3* internet6 stream tcp c4443000
root     dropbear     637    4* internet stream tcp c4442cb0

Inode 236554 is (thanks double_p :)
Code: 
236554     1152 -rwxr-xr-x    1 root             wheel              570988 Nov 29  2010 /usr/include/php/dropbear
Code: 
# ls -l /usr/include/php/dropbear
-rwxr-xr-x  1 root  wheel  570988 Nov 29  2010 /usr/include/php/dropbear
oOo...
Code: 
# file dropbear
dropbear: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 6.4, statically linked, FreeBSD-style, stripped
Code: 
# md5 dropbear
MD5 (dropbear) = 02a9e508a758d5fcf8822f957af8fc6a
Strings: http://pastie.org/4790612

If anyone has more input to this, I would appreciate. Thanks.
 
ernie said:
I find that when you do an upgrade, any hacked binaries get overwritten, mergemaster should point out any diffs.
Click to expand...

This is months old, but still needs comment.

It's not safe to trust anything on a compromised system. That includes the compiler, or install(1), mergemaster(8), or any other component involved in rebuilding the kernel or world. Here is an article that describes what Ken Thompson did many, many years ago: http://scienceblogs.com/goodmath/2007/04/15/strange-loops-dennis-ritchie-a/.

Even if you boot from another system and then reinstall over the compromised one, it is very hard to be sure that all the compromises are fixed. That's why the general recommendation is to back up data, wipe the system, install from scratch, and then restore just data.
 
You should also run a bios update from the manufacturer as well as an update for any "Lights Out"/Service processor, if you're paranoid, which maybe you should be?
 
... because, not being paranoid does not mean that T.H.E.Y. are not out to get you.
There have been proof of concept hacks of keyloggers in the keyboard controllers. So the question always is, what can you trust in any case.

Good article from Ken, BTW. Definitely a good read and also something you want to have around when it is said that some anti virus software is a cure-all.
 
Perhaps offtopic, but to really ensure the compiler was trusted you'd have to write it yourself in assembly on an air-gapped system, and use that to bootstrap gcc's C sources. Yikes. Or is there an easier way to ensure compiler purity?
 
wblock@ said:
This is months old, but still needs comment.

It's not safe to trust anything on a compromised system.
Click to expand...

This. Nuke it from orbit, it's the only way to be sure.



Q: why are you still running 6.x or 7.x? Sooner or later you're going to get owned due to the lack of support for a release so old.

Rather than waiting for that to happen you should be doing your best to migrate to a supported release ASAP.
 
You must log in or register to reply here.
Back
Top