Postfix

The data of important files to be saved can be scanned for viruses.

It's the operating system files that have to be wiped clean.

Unless it's a problem of a server with many computers, then it's like athletes foot. Perhaps if there's many computers, take them all offline, then put up one new installed operating system at a time, modularly. Someone else will have to give you advice on that. You may need to check that infected harddisk mounted as read-only to get an idea of where it came from, to prevent it better.

Also, if you know which port it came in through, use the firewall pf, ipfw and/or ipf to block network activity on that port if it's not needed for the future.

IPFW can be turned on by setting this in rc.conf:
Code:
firewall_enable="YES"
firewall_type="workstation"
# "server", "client" or other settings can also be used here.
# Only if these custom firewalls allow the traffic that you need

Then you can have an additional firewall of PF or IPF, to lock down more on it.
 
It is not possible to create these websites from scratch.
Those are some of the important files you need to save and backup then.

Definitely save those infected harddrives, and get new ones. You'll need those old hard-drives mounted as read-only in case anything was missed that needed to be saved, as for important files, configuration files and website data to use.

SSD drives are cheap ($20), and they can be used for paritions that (only or mostly) contain the operating system. Then have important files in partitions in another hard-disk that's reliable, then you can virus-scan that, and put it up to be used with a new or reinstalled operating system. It's modular to have the OS on one drive, then other files on another drive.

But important files need to be backed up, before messing with formatting, switching out harddrives and reinstalling. The old infected harddrive can serve as another backup for anything missed, and for investigating. Perhaps label it on top with a marker.
 
You advise me to create 30 virtual servers for each site separately to determine which site contains the virus.
 
I don't think so.

But the problem looks difficult, that you'll need someone else to help you better, or you'll figure out what steps to do, that you'll save important configuration files, server data, website configurations and other important files. Sometimes you'll come back to a problem, and be able to solve it better.

The Operating System is mostly what needs to be started from new. Then website data, virtual server configurations, all configurations, all website data, etc needs to be virus-scanned, and important files saved.

That virus intended to mess with something difficult as I see by reading this, but you will figure it out, perhaps with additional help.
 
How did you use Clamscan?

You can use many operating systems to mount that harddrive as read-only. Save/backup the files that weren't infected. Save that old hard-drive with its data, because you'll likely need to extract information and data later.

On a new install, perhaps have the OS on a separate harddisk than the non-OS files. Careful as Windows sometimes wants to delete filesystems, if it doesn't understand the existing FreeBSD or Linux filesystem.
Then, you can mount that harddrive in the future as needed from many OS's. Also that OS harddrive can be wiped clean each time that's needed, so long as you take precaution to identify the harddrive with the OS and one with important data, and back up that data as well.

I would say, only new 2 harddisks are needed for this, plus save the old one(s). Maybe a 20 or 40G SSD SATA drive for the OS, and then a traditional SATA drive that has as many giga/terabytes as needed.

I read that old infected harddisks can be taken in for forensics to find out who did what, but you'll also need it for important files, which may be missed, or to look into corrupted files that you needed.

Buy a refurbished basic motherboard if needed for a new computer, as it costs less.
 
I checked ClamAV and rkhunter. There are no more viruses.

That's great, now migrate the entire system to a new install like others have mentioned here. Get a completely new computer if you can't take this one down, do it on the weekend, something... sheesh.
 
That's great, now migrate the entire system to a new install like others have mentioned here ...
This means that the ClamAV could not find viruses and these viruses are there at the moment. I don't want to migrate these viruses to a new server.
 
Mount that harddrive as readonly, from an operating system that has a good virus scanner.

Windows or Linux may get it, but it may not understand the filesystem and try to delete it. There should be another way to get it mounted and scanned from FreeBSD or Linux.

The problem looked mostly to be in Operating System and port's processes. It's a set of executable (and perhaps other) files working together. The problem is from files running, that are hidden that act like what you found in htop, not likely from something like ".mp3", unless that's used to hide something, and that wouldn't be made obvious as suspicious by that being in a bin/ directory. It's like something from a bin/ directory or rootkit. There's some distinction from operating system, ports and executable files than important non-executable files.

Needed files (text files, configuration files, sound files, picture files) don't usually run. Viruses/malware can come hidden as mp3, txt or jpg files. You can also set certain partitions to be as non-executable, where most important files belong anyways. Non-executable is supposed to block files within its partition from running.

You may have to do some reading on the subject.
 
If the viruses and/or backdoors are in your content, then you've got a security hole in your applications and the whole thing should be shut down and audited. If you're not going to do that then....

If the viruses are in your OS, then migrating the sites to a clean install & known secure configuration with new keys, passwords, etc, will solve the problem.
 
It is not possible to create these websites from scratch.
I'm sorry but this has to be nonsense.
I don't know your government but I do lots of work for our federal government & if our company let this go we'd be blackballed from future tenders. Period.

You need to do as all the others are advising you to do. If you don't have the authority to do it, advise someone who does.

Anti-virus software is a security placebo. You need to secure your systems. You need to find the source of access and stop it. You need to backup your data. You need to completely audit your software. No sense rebuilding the server & putting the same, insecure software on it.

Obtain another server or instance and export only your data. Change all database passwords. Audit your data. Install the cleaned/audited software, monitoring software etc.

I'm telling you nothing new as all the other posters have told you. I'm just adding to the chorus urging you to take drastic action now because if you don't imagine when you're server's totally controlled by someone and they lock you out, what will you do then?
 
Whatever it took to make those websites, back that up, and put it back in, so that it's not from scratch.
It may be a language thing, but it's hard to get a clue on what's running on this server. Maybe I missed something? Is it custom PHP or some package or port?
It's not immediately certain whether the problem is the OP is just out of their league with this. (No shame in that, I might add).
 
akshin just said websites. Whatever it is, that's data that can be backed up, whether html, php, http configurations, or anything else.

akshin may have to call someone they know for help and to offer a job to.
 
But how? This is problem. There is no antivirus for unix and linux.
make -C /usr/ports search key=virus | egrep '^(Port|Info):' | less tell us there are two freely availabe virus scanners, security/f-prot & security/clamav and a bunch of ports to integrate these into mail etc. You told before and after this post you used clamscan to detect infected files...
You need to do as all the others are advising you to do. If you don't have the authority to do it, advise someone who does. [...] Anti-virus software is a security placebo. You need to secure your systems. You need to find the source of access and stop it. You need to backup your data. You need to completely audit your software. No sense rebuilding the server & putting the same, insecure software on it. Obtain another server or instance and export only your data. Change all database passwords. Audit your data. Install the cleaned/audited software, monitoring software etc.
Nothing to add to that. Except two topics:
  • you may want to call your government's CERT team to help you, they can provide you a recipe of actions to follow.
  • Please be aware that loosely speaking of ipfw(4) (or pf(4) & ipf(4)) as a firewall is not strictly correct, these are packet filters and are vital parts of a firewall. You (your team) may need to review your firewall setup, and foremost your web application design. Obviously someone broke in and put the virus on your site. You need to fix that hole before going online again.
 
Trying to remove these viruses is an exercise in futility. This is highly customizable code, scanners will not be able to detect them. Besides that, this machine has been compromised, everything you run on it is tainted and cannot be trusted.

Note that these bots likely came in through a code-injection in some bad PHP code. Packet filters aren't going to protect you against this type of attack.
 
Having a positive break in on such a webserver means: Use the source of your PHP webpage - and never ever copies of the files from the compromised machine (they could have been modified!). If it's a PHP driven website, then there has to be someone who has an equivalent development machine from which this rebuild can be done. If that "master computer" doesn't exist: Huge fail and dead loss; But even a backup can only be used if you're sure it has been taken before the break in.

And before the server is getting up & running again you've got to find out how the break in happened: Mustn't been PHP, could also be a weak SSH account etc. But if it was PHP: The programmers should do some homework before (and never ever set up a PHP site and forget or just expand it - continuous security checks of the code are non-optional, dynamic websites means work every single month). To get a clue of this "how" take a deep look in all of the logfiles (but if "they" were good you won't find anything). Otherwise it won't take long till this happens again.

On a server there's nothing to "clean up and reboot" - this installation has to be canceled. Really.

And by all worse and upcoming work - if setting up the server cannot be done: None server is that important that live wouldn't be able anymore, so : shut it down, learn your lessons, go deeply over your concept, find a short term solution and start a better project.
 
In the long run, get rid of PHP.
That's a bit silly. It's not the language that's at fault, but the programming of it. Sure PHP has had a history of exploitation but, boy oh boy, have some programmers just got no idea! The copy/paste programmers just hope for the best. :rolleyes:

And replace it with what? Perl? Python (good until their mighty overlord decides to break everything... AGAIN).
PHP is fine, just hope the programmer has a clue. ;)
 
That's a bit silly. It's not the language that's at fault, but the programming of it. Sure PHP has had a history of exploitation but, boy oh boy, have some programmers just got no idea! The copy/paste programmers just hope for the best. :rolleyes:

And replace it with what? Perl? Python (good until their mighty overlord decides to break everything... AGAIN).
PHP is fine, just hope the programmer has a clue. ;)
PHP is a bad choice for a principle reason: a commonly accepted guideline in software engeneering states: do not mix application logic & UI appearence logic. PHP violates this -- by design, i.e. this flaw is inherent in PHP. This way of programming appeals hackers (in the sense of quick & dirty hack, not: break into a system), i.e. it misleads to do "dirty" programming. Yes, you can mess up your software in every language. It's just much easier in PHP. Compare the impressive list of security alerts of PHP to e.g. Plone (framework Zope, language Python).
 
Back
Top