Recent botnet installed as root

Indeed. Doing the upgrade dance over several versions is no pain at all when compared to wiping the system completely and reinstalling followed by pulling user data from backups and in the meantime apologizing to every user for the downtime.
 
That's why a lot of admins work off-hours and weekends.
 
throAU said:
Q: why are you still running 6.x or 7.x? Sooner or later you're going to get owned due to the lack of support for a release so old.
Click to expand...
I'm running 6.4 on 3 systems (and 8-STABLE on about two dozen other systems). Two of the 6.4 systems are running extremely modified versions of RANCID, MRTG, and NAGIOS (among other things) and I've been making a half-hearted attempt to get some of my mods accepted by the upstream maintainers. Some of them will never accept the changes as they're specific to the way my company generates an all-in-one customer dashboard. So there's a fair amount of stuff which will probably need to be rewritten (or at least new-from-scratch patches created). Another example is the IPMI-over-PPP that these older boxes use. There doesn't seem to be a way to get that to work, given the change away from Kernel PPP and a bunch of changes to the uart / sio / puc driver set. The 3rd 6.4 box is dedicated to a customer who is planning on merging whatever he's doing on that box to a new 8-STABLE box.

I do have the 3 replacement systems (all Dell PowerEdge R300s) installed with 8-STABLE and a bunch of the back-end processing is happening on them. Unfortunately, some of the 3rd-party software we use has a nasty habit of rewriting URL's into dotted-quad form, so it isn't as easy as it would seem to be to move pieces of the critical software over. Then there's things like several hundred Cisco routers sending us syslog data via IP address instead of hostname (some of those routers are not controlled by us, and getting them to change their IOS config to log to a different host is next to impossible).

So, this pretty much means a hopefully-sufficient level of pre-deployment regression testing, followed by a forklift upgrade of the whole system. Which is made a bit more problematic as at the same time we're doing this, we're also moving to a new colocation site.

Rather than waiting for that to happen you should be doing your best to migrate to a supported release ASAP.
Click to expand...
I'd be perfecly willing to take responsibility for keeping the kernel and base patched, but what has really set me back quite a bit with ports is the intentional breakage* of the new ports build structure. While I could keep most of my 6.4 ports up-to-date that way, the new ports build structure just throws a lot of errors and quits. I could even understand ports that were tweaked for options-ng not being buildable on old boxes, but I can't even build a port that was already installed before the Great Ports Breakage and which hasn't had its port Makefile and so on changed.

* If you look at the commits for files in /usr/ports/Mk, you'll see a bunch of commits to fix backward compatibiliy, but they were backed out and not resubmitted.
 
Backward compatibility on the ports system for 6.x systems won't happen, there's no support for them anymore.
 
Hello Everyone.
I had the same problem. I found on my FREEBSD 2 files, "talkngbsd" and "dnskill4bsd", both installed as root / wheel. They was on /root folder. Anymore, I had several daemons working named "barbut2". if killed, they keep on working. I had to delete the 2 files, talkngbsd" and "dnskill4bsd" and then reboot. I have the 2 files i could post them somewhere 4 analysis. Now it seem to work fine, BUT I can't understand HOW DAMN they succeded. i don't have telnet working, maybe sendmail ? does anyone knows how they did the exploit?
 
Terry_Kennedy said:
I'm running 6.4 on 3 systems (and 8-STABLE on about two dozen other systems). Two of the 6.4 systems are running extremely modified versions of RANCID, MRTG, and NAGIOS (among other things) and I've been making a half-hearted attempt to get some of my mods accepted by the upstream maintainers.
Click to expand...


Oh I get it, keeping updated is painful. I don't like having to upgrade either. I don't think any of us like having to mess with production boxes that are doing their job.

However, your choices are:
- backport security updates to 6.x yourself
- upgrade to a supported release
- run the gauntlet without patches and eventually get owned.

For most people, option 2 is the path of lease resistance over the long term. Option 3 is easy over the short term, but turns into a lot of work that needs to be performed under duress eventually.

Your company needs to decide whether or not the massively customized environment you have is worth the support cost, and if so, allocate enough resources (staff) for it.
 
Hi Never,

I posted that same issue back in March. I was turned off by the responses; or the lack of any constructive data/response. I never did get an answer or insight onto the attack vector. I am still very curious. (My first nix hack.) I did take an image of the box and have it somewhere. From my understanding of hacking, I wonder if there is more value in pursuing such a database and analysis (?)... which is what I was/am seeking.

Anyway, people here seem to keep harping on the obvious... beating you up over why don't you keep patched and updated, etc, etc. Focusing on WHY you were hacked and not HOW. AVOIDING my question of does anyone KNOW anything about those files, the attack vector, etc?? Any such technical analysis is VERY welcome.

I just get the CLEARLY OBVIOUS... do not trust a compromised host, wipe, re-install, install latest software and keep patched. Ok, I agree, but that was not my question. I know I am running old code. Other than the code, I thought the box was fairly well hardened. I know it's good to patch, and to install the latest builds, but this was a 'hobby' box, I am not a corporation, etc, etc. I just don't have time to keep maintaining systems all day.

I know the data exists, but I'm not sure where to start to be bluntly obvious.... I just don't have much spare time to invest now... I don't know where to find a list of all vulnerabilities for my daemons (what patches are available for me to install, etc), and possibly even cross-reference that with reports of exploits in the wild, signatures, other reports. These files are unique... Google doesn't have a lot of matches... what are the databases / repositories of MD5's of root kits, virus, malware binaries, hacks, etc...? And a database of 'case fies' ... I had X files, this was my system config, and if known, this is how they got in ... be it via a service, even if the exact exploit used on that service was not known, etc...???
Any hobbyists do forensics and keep such data?

I'll summarize my setup:
FreeBSD 7.2-RELEASE (unpatched, unmaintained since install) (release date May 1, 2009)
DNS (bind9: $FreeBSD: src/etc/namedb/named.conf,v 1.26.2.2.4.1 2009/04/15 03:14:26 kensmith )
SSH (OpenSSH : FreeBSD-20080901)
NTP
TCP 3130 (header: "SSH-2.0-WS-FreeBSD" ... not sure what this is...?)
Telnet (firewalled to only LAN access)

no sendmail, no Apache, root SSH access disabled

I can say that my netflow, firewall and local and remote syslog did not produce any unknown SSH logins... it's just one user (me)... so I'd say I'm 90% sure that they did not SSH into the box to load the code.
 
inetd.conf - only SSH & telnet v4 is enabled
(external ACL that blocks telnet to all internal hosts + syslog that sends to remote syslog all unsuccessful and successful logins via telnet and SSH ... these logs show only me getting in successfully, as do all local logs, as do all file timestamps, etc)
 
throAU said:
Oh I get it, keeping updated is painful. I don't like having to upgrade either. I don't think any of us like having to mess with production boxes that are doing their job.

Your company needs to decide whether or not the massively customized environment you have is worth the support cost, and if so, allocate enough resources (staff) for it.
Click to expand...
As I mentioned, I have some two dozen boxes running 8-STABLE, which get cvsup'd (and any updated ports built) at least weekly, and which get new kernels monthly.

The big problem is the customized stuff on the three 6.x boxes. Like many businesses, we have some old code. Would it be good to re-write it from scratch? Probably. Does that make any sort of business sense at all? No.

However, your choices are:
- backport security updates to 6.x yourself
- ...
Click to expand...
My big issue here is that ports were intentionally broken and not fixed for 6.x. If you look at the commits from that time period, you'll see that there were a bunch of commit comments about "fix compatibility breakage with old infrastructure" and so on. Unfortunately, as those commits went in, the level of breakage increased.

I know that this is an unsupported release. However, it is very rare that FreeBSD just discards compatibility without caring. In particular, I am unaware of any kernel / ABI functions necessary for the ports infrastructure which is available in 7.x and newer but not in 6.x. So why doesn't it work? Out-of-date ruby interpreter? Something else?

It doesn't help that I'm not enamored of the new ports system, either - ports which were configured and built previously throw up a new config screen (thus halting an automated portupgrade -a). This is true even when the options haven't changed on the port - apparently this is a "feature" of the new ports options - when a port is converted to the new options structure, the user gets a configure screen again. Whether or not the values shown are the values the user selected in a prior configure process or some other set of default values seems to be at the whim of the port maintainer, or perhaps the phase of the moon.

Next, there have been changes to the new global options strings, which means that one day a port will describe the "WITH_FOO =" option with one string, and the same port built on a different day on a different system will use a completely different string to describe the same option.

The only issue I ever had with the old ports system was that upgrading some ports required a manual restart of the service, as the port build / install killed the running instance. It would be nice to have a "The following ports may need to be manually restarted:" section added to the bottom of a portupgrade -a run, listing those ports where the port's process(es) were running but were killed during the upgrade.

But I'm probably talking to an nearly-empty room about this. There was a whole ruckus when FreeBSD 4.x (IIRC) was EOL'd, causing lots of discussion and the creation of the freebsd-eol[i]@[/i]lists.freebsd.org list (which has a grand total of two non-spam posts since it was created nearly 6 years ago.
 
At the end of the day, my post wasn't about the politics involved - as far as I'm concerned as an end user, the politics are irrelevant. It was from a pragmatic standpoint (I'm not a developer).

The situation is what it is. Your choices are to keep up to date with a supported release, backport yourself, or get hacked. If none of those options are suitable, then you need to consider shifting to another platform with a longer support window (again, this is me being pragmatic, I'm not a FreeBSD developer and don't speak for the FreeBSD team, etc).


As to discovering HOW it was hacked (what vector) - that isn't something anyone here on this forum can help you with really. There are a number of holes in 6.x and probably in the third party software you also have heavily customized.


edit:
Looking at the list of software above - pretty sure there's a remote exploit for Bind from a year or two ago, let alone from 2009.

A complete list of Bind security advisories available here:
https://www.isc.org/advisories/bind
 
Hi Kleinart,
I DO agree what you say: I know that if everything is updated every world is more secure, and in this discussion I also see what to do after and not HOW they did. I think that for our nature "hacking, in good way", would be nice if we succeeded in duplicating the problem. I made some research, and I found an exploit about openSSH: http://www.securityaegis.com/openssh-0day/ and hope to have time to try. Anyway,here the list of all packages installed. I have 6.4 FreeBSD release
Code: 
ImageMagick-nox11-6.7.5.10 Image processing tools
OpenSSH-askpass-1.2.4.1 Graphical password applet for entering SSH passphrase
alpine-2.00_3       Mail and news client descended from Pine
apr-devrandom-gdbm-db42-1.4.5.1.3.12_1 Apache Portability Library
arc-5.21p           Create & extract files from DOS .ARC files
arj-3.10.22_4       Open-source ARJ
aspell-without-dicten-0.60.6.1_1 Spelling checker with better suggestion logic \than ispell
autoconf-2.13.000227_6 Automatically configure source code on many Un*x platfor\ms
autoconf-2.68       Automatically configure source code on many Un*x platforms
autoconf-wrapper-20101119 Wrapper script for GNU autoconf
automake-1.4.6_6    GNU Standards-compliant Makefile generator (1.4)
automake-wrapper-20101119 Wrapper script for GNU automake
bash-4.2.24_1       The GNU Project's Bourne Again SHell
bison-2.5,1         A parser generator from FSF, (mostly) compatible with Yacc
bitstream-vera-1.10_5 Bitstream Vera TrueType font collection
bsdpan-Mail-SpamAssassin-CompiledRegexps-body_0-1.0 Mail::SpamAssassin::Compile\
dRegexps::body_0 - Efficient str
ca_root_nss-3.13.4  The root certificate bundle from the Mozilla Project
cclient-2007f,1     Mark Crispin's C-client mail access routines
cdialog-1.1.20111020,1 An enhanced version of 'dialog' to work with ncurses
clamav-0.97.4       Command line virus scanner written entirely in C
compat4x-i386-5.3_9 A convenience package to install the compat4x libraries
compat5x-i386-5.4.0.8.1_1 A convenience package to install the compat5x librari\
es
compositeproto-0.4.2 Composite extension headers
cs-aspell-20040614.1_1,1 Aspell Czech dictionary
curl-7.24.0         Non-interactive tool to get files from FTP, GOPHER, HTTP(S)
cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI v\
ersion
cy-aspell-0.50.3_1,1 Aspell Welsh dictionary
cyrus-sasl-2.1.25_2 RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-saslauthd-2.1.25 SASL authentication server for cyrus-sasl2
da-aspell-1.4.42.1_1,2 Aspell Danish dictionary
damageproto-1.2.1   Damage extension headers
darts-0.32          A C++ template library that implements Double-Array
db4-4.0.14_1,1      The Berkeley DB package, revision 4
-=--:----F1  lista.txt      Top L1     (Text)-----------------------------------
For information about GNU Emacs and the GNU system, type C-h C-a.
File Edit Options Buffers Tools Help
ImageMagick-nox11-6.7.5.10 Image processing tools
OpenSSH-askpass-1.2.4.1 Graphical password applet for entering SSH passphrase
alpine-2.00_3       Mail and news client descended from Pine
apr-devrandom-gdbm-db42-1.4.5.1.3.12_1 Apache Portability Library
arc-5.21p           Create & extract files from DOS .ARC files
arj-3.10.22_4       Open-source ARJ
aspell-without-dicten-0.60.6.1_1 Spelling checker with better suggestion logic than ispell
autoconf-2.13.000227_6 Automatically configure source code on many Un*x platforms
autoconf-2.68       Automatically configure source code on many Un*x platforms
autoconf-wrapper-20101119 Wrapper script for GNU autoconf
automake-1.4.6_6    GNU Standards-compliant Makefile generator (1.4)
automake-wrapper-20101119 Wrapper script for GNU automake
bash-4.2.24_1       The GNU Project's Bourne Again SHell
bison-2.5,1         A parser generator from FSF, (mostly) compatible with Yacc
bitstream-vera-1.10_5 Bitstream Vera TrueType font collection
bsdpan-Mail-SpamAssassin-CompiledRegexps-body_0-1.0 Mail::SpamAssassin::CompiledRegexps::body_0 - Efficient str
ca_root_nss-3.13.4  The root certificate bundle from the Mozilla Project
cclient-2007f,1     Mark Crispin's C-client mail access routines
cdialog-1.1.20111020,1 An enhanced version of 'dialog' to work with ncurses
clamav-0.97.4       Command line virus scanner written entirely in C
compat4x-i386-5.3_9 A convenience package to install the compat4x libraries
compat5x-i386-5.4.0.8.1_1 A convenience package to install the compat5x libraries
compositeproto-0.4.2 Composite extension headers
cs-aspell-20040614.1_1,1 Aspell Czech dictionary
curl-7.24.0         Non-interactive tool to get files from FTP, GOPHER, HTTP(S)
cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI version
cy-aspell-0.50.3_1,1 Aspell Welsh dictionary
cyrus-sasl-2.1.25_2 RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-saslauthd-2.1.25 SASL authentication server for cyrus-sasl2
da-aspell-1.4.42.1_1,2 Aspell Danish dictionary
damageproto-1.2.1   Damage extension headers
darts-0.32          A C++ template library that implements Double-Array
db4-4.0.14_1,1      The Berkeley DB package, revision 4
db41-4.1.25_4       The Berkeley DB package, revision 4.1
db42-4.2.52_5       The Berkeley DB package, revision 4.2
dirmngr-1.1.0_8     A client for managing and downloading certificate revocatio
dmxproto-2.3.1      DMX extension headers
dovecot-1.2.17      Secure and compact IMAP and POP3 servers
e2fsprogs-libuuid-1.42.2 UUID library from e2fsprogs package
el-aspell-0.50.3_1,1 Aspell Greek dictionary
emacs-nox11-23.4_8,2 GNU editing macros
en-aspell-7.1.0     Aspell English dictionaries
encodings-1.0.4,1   X.Org Encoding fonts
es-aspell-1.11.2,1  Aspell Spanish dictionary
 
Never said:
I have 6.4 FreeBSD release
Click to expand...
This version has been End-of-Life since November 2010 and contains quite a number of security vulnerabilities.

If you want to know who got in and how I suggest you take it to a forensic investigator.
 
Well, @kleinart. Post a binary or do your own analisys. Don't assume that we will explain with no binary at all (in fact, it's actually better if you give it a shot by yourself. If you don't know the tools, you could ask).

Analyzing what syscalls are used, with what, which files are touched, what is written or read, what network traffic comes up. Could be a good start.

Regards.
 
m6tt said:
If you have Linux boxes out there, you may want to check on them...seems like bad news.
Click to expand...

If you have ANY boxes (regardless of operating system) "out there", you need to be keeping them up to date. Be it Linux, FreeBSD, Windows, etc.

Dealing with potential zero-days is bad enough; if you are running software that has had well-known vulnerabilities published for several years there's little point even bothering to do forensic analysis IMHO. There are plenty of known holes; which one an attacker used isn't really relevant unless you're going to fix it and also close the other known-for-years X holes left in your un-patched machine.


The "we have so much custom stuff!" isn't really an excuse. Yeah it sucks, but it doesn't change the reality you face.

I know posters here may not be the decision maker in the company, but it is your duty to stress to management that if they want to run some super custom solution, the cost of ownership does not stop when the software is deployed. Resources MUST be allocated to keep the software up to date as required, or eventually you WILL get owned.

If the company is not willing to allocate resources for this, then the software is un-supportable and should be decommissioned, unless management acknowledge the risk and accept it. It may not be what management WANT to hear, but far better that they know about the issues involved and then make a calculated business decision ON THEIR OWN HEAD to continue running an un-supportable platform. At the end of the day if fixing the problem is beyond the resources you have available, it is a business/financial decision to make and they will either spend the money on additional resources to mitigate the risk or they won't.

However, if you haven't alerted management to the problem, then you have essentially taken on responsibility for the security of your boxes yourself and IMHO are placing yourself personally at risk for the fall-out, should they get owned. Don't be the fall-guy.
 
throAU said:
If you have ANY boxes (regardless of operating system) "out there", you need to be keeping them up to date. Be it Linux, FreeBSD, Windows, etc.
Click to expand...
Couldn't agree more. Unfortunately there are a lot of people out there that think OS A is better than OS B and they don't need to secure their stuff. Because they are in the mistaken belief there's something magical called unix that will prevent any and all infections or hacks.
 
m6tt said:
I do wonder.

I recently started hearing about an attack with a similar MO affecting Linux machines:
http://www.webhostingtalk.com/showthread.php?t=1235797&page=38
http://www.reddit.com/r/netsec/comments/18ro3c/sshd_rootkit/

I immediately thought of the variety of machines affected and mentioned in this thread.

If you have Linux boxes out there, you may want to check on them...seems like bad news.
Click to expand...



I tried to read trough the linked material but I couldn't find anything definite that would point to a vulnerability in sshd(8). Lots of speculation whether this is yet another PHP vulnerability though...
 
It's not related to sshd(8), that's just one of the symptoms. It's unclear how exactly they got in but my guess is some broken web application. They then used this 0-day Linux kernel bug to gain root and backdoor sshd(8).

http://blog.solidshellsecurity.com/2013/02/08/sshd-spam-rootkit-lib64libkeyutils-so-1-9/

Obviously just removing that library and restarting sshd(8) doesn't solve the issue it only removes the symptoms. Because the attackers had unrestricted root access nothing on that server should be trusted anymore.
 
I agree it's very unlikely sshd...I think the last commonly exploited remote in sshd was back in the 90s/Early 00s?

If it was sshd, the entire internet would be pretty messed up right now :)

I did see some folks suspecting that their admin workstations had been owned, and that keyloggers/data disclosures on Windows & Mac boxes had been used to gain access. Makes some sense when thinking about the recent Java and Flash vulnerabilities...just a reminder to keep your administration tools as secure as your servers, or else it doesn't matter how fancy your security is if an adversary can piggyback.
 
SirDice said:
Couldn't agree more. Unfortunately there are a lot of people out there that think OS A is better than OS B and they don't need to secure their stuff. Because they are in the mistaken belief there's something magical called unix that will prevent any and all infections or hacks.
Click to expand...

Exactly, unfortunately.

Sure, you can reduce your exposure somewhat by going for a less popular or more cut down OS, due to the number of vulnerabilities found - but if a vulnerability is found, you need to fix it or mitigate it, regardless of OS.
 
Terry_Kennedy said:
The big problem is the customized stuff on the three 6.x boxes. Like many businesses, we have some old code. Would it be good to re-write it from scratch? Probably. Does that make any sort of business sense at all? No.
Click to expand...

Yes it does.

That business case is called "restart after being owned by evil hackers", and has to be payed in downtime and PR budget. More often than not this case is closely linked to something called "chapter 11", so it does not seem to be a particular profitable case. And after all, it's a case which is not initiated by your company but someone else.
 
throAU said:
Exactly, unfortunately.

Sure, you can reduce your exposure somewhat by going for a less popular or more cut down OS, due to the number of vulnerabilities found - but if a vulnerability is found, you need to fix it or mitigate it, regardless of OS.
Click to expand...


No one can really argue with any of these comments. But they're also not really constructive to this thread, IMO.

In my case, I'm fairly sure DNS/bind was my attack vector. My box was a hobby box, running SSH (locked down, and for my remote shell access/bastion host) and DNS. Being that it was so limited, I didn't think much of it. Lesson learned.

Agree, as a business - if you have assets, you need to protect them appropriately. AND, if you are compromised, the appropriate response is warranted. And, you should fully rebuild a compromised system (regardless of who you are); I did. Professionally, yes you need to maintain your stuff. Hobby systems - one person managing 15+ home systems - easier said than done.

What will NOT happen, as things currently are, will be all systems patched immediately - or even maintained perfectly. In direct response to the above quote ... there are vulnerabilities found daily, weekly and monthly ... and within a 'short time' you may not have immediate patches available - your dist may not be maintained and you need to do major work to upgrade your system. Or, in other words, patches probably take too much time and effort as-is, and in the future will probably be more automatic and self-protecting systems and code - just needs evolution time. (Windows and other desktop OS's - OSX, etc have come far in this regard - automatic updates, etc.) In this case for me, patches weren't available - I had to upgrade the major revision of the OS... at the time I just figured that was too much effort as I didn't perceive a risk to the box. I was quite surprised the box got owned - with its limited daemon exposure.

// Wish more effort was put into identifying and classifying root kits, tacking trends, tracking down countries they come from, prosecuting as possible, etc, etc. Probably hugely controversial - and is currently a profit item ... but non-profit / for the benefit of everyone ... free systems that probe for vulnerabilities and can notify their owners, etc. Much like I can sign up to receive spam/abuse notices from my IP space, etc. If hackers can find me, why not whitehats find me first and try to be proactive and help?
I personally already blackhole all of China and N Korea, and will probably add Russia to my list. Personally, on my hobby net, I have no legitimate reason to really be communicating with those countries, plus I find them to be a risk and not responsible in maintaining law/order/security... let alone the spam issues that had existed (lesser lately than in the last few years).
 
You must log in or register to reply here.
Back
Top