FreeBSD expert in webserver config needed

[duce]

Good day,

Could I ask that someone feeling like doing some freelance work contact me please.
You need to know pf, postfix, apache (http and https), mysql and kernel config.

There might also be the need to upgrade the box from FreeBSD 8 to the latest stable version.

Lately my server has picked up on load averages of over 40.00 60.00, etc. CPU constantly over 60% and I am simply not finding the issue no matter how much I Google.

Thanks,
Paul

 

[SirDice]

If the server is that old and is showing a really high load it's probably been infected with malware.

 

[obsigna]

If the server is that old and is showing a really high load it's probably been infected with malware.

More information would be useful, e.g. which process(es) impose(s) so much load to the CPU(s)
Please submit the command ps -axj on the console and post the result.

Are you looking for online assistance, e.g. via ssh, or do you need on-site assistance?

 

[duce]

SirDice - Did not think of that. Busy running a scan. Thanks.
obsigna - Support via ssh.

USER      PID  PPID  PGID   SID JOBC STAT  TT       TIME COMMAND
root        0     0     0     0    0 DLs   ??    0:02.89 [kernel]
root        1     0     1     1    0 ILs   ??    0:00.05 /sbin/init --
root        2     0     0     0    0 DL    ??    0:27.82 [g_event]
root        3     0     0     0    0 DL    ??   54:24.68 [g_up]
root        4     0     0     0    0 DL    ??   38:53.25 [g_down]
root        5     0     0     0    0 DL    ??    0:00.00 [sctp_iterator]
root        6     0     0     0    0 DL    ??    0:05.81 [pfpurge]
root        7     0     0     0    0 DL    ??    0:00.00 [xpt_thrd]
root        8     0     0     0    0 DL    ??    7:56.42 [pagedaemon]
root        9     0     0     0    0 DL    ??    2:42.54 [vmdaemon]
root       10     0     0     0    0 DL    ??    0:00.00 [audit]
root       11     0     0     0    0 RL    ??  2247:50.40 [idle]
root       12     0     0     0    0 WL    ??   60:51.89 [intr]
root       13     0     0     0    0 DL    ??    3:42.66 [yarrow]
root       14     0     0     0    0 DL    ??    0:05.36 [usb]
root       15     0     0     0    0 DL    ??    0:00.01 [pagezero]
root       16     0     0     0    0 DL    ??    0:02.55 [bufdaemon]
root       17     0     0     0    0 DL    ??    0:28.13 [vnlru]
root       18     0     0     0    0 DL    ??   14:14.77 [syncer]
root       19     0     0     0    0 DL    ??    0:41.83 [softdepflush]
root       20     0     0     0    0 DL    ??    0:10.01 [flowcleaner]
root      479     1   479   479    0 Is    ??    0:00.01 pflogd: [priv] (pflogd)
_pflogd   489   479   479   479    0 S     ??    0:13.32 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd)
root      636     0     0     0    0 DL    ??    0:00.21 [accounting]
root      949     1   949   949    0 Ss    ??    2:02.69 /usr/local/bin/spamd -u spamd -H /var/spool/spamd -d -r /var/run/spamd/spamd.pid (perl)
spamd     952   949   949   949    0 I     ??    0:00.87 spamd child (perl)
spamd     956   949   949   949    0 I     ??    0:01.07 spamd child (perl)
root     1531     1  1531  1531    0 Is    ??    0:03.16 /usr/sbin/sshd
openvpn  1623     1  1623  1623    0 Ss    ??    0:20.97 [openvpn]
openvpn  1630     1  1630  1630    0 Ss    ??    0:10.33 [openvpn]
openvpn  1635     1  1635  1635    0 Ss    ??    0:13.18 [openvpn]
root     1673  1531  1673  1673    0 Ss    ??    0:19.51 sshd: duce@pts/8 (sshd)
root     3214  1531  3214  3214    0 Is    ??    0:00.16 sshd: duce@pts/12 (sshd)
vscan   16416     1 16416 16416    0 Is    ??    2:16.92 /usr/local/bin/freshclam --daemon -p /var/run/clamav/freshclam.pid
vscan   17095     1 17095 17095    0 Is    ??    2:01.65 /usr/local/bin/freshclam --daemon -p /var/run/clamav/freshclam.pid
vscan   19838     1 19838 19838    0 Is    ??    1:45.08 /usr/local/bin/freshclam --daemon -p /var/run/clamav/freshclam.pid
root    31942     1 31942 31942    0 Ss    ??    0:59.62 screen
root    34226  1531 34226 34226    0 Is    ??    0:00.30 sshd: duce@notty (sshd)
duce    34227 34226 34227 34227    0 Is    ??    0:00.09 -zsh (zsh)
vscan   34302     1 34302 34302    0 Is    ??    1:15.58 /usr/local/sbin/clamd
root    34333     1 34332 34332    0 I     ??    0:00.00 /usr/local/sbin/syslog-ng -p /var/run/syslog.pid
root    34334 34333 34334 34334    0 Ss    ??    0:01.34 /usr/local/sbin/syslog-ng -p /var/run/syslog.pid
root    34354     1 34353 34353    0 S     ??    1:57.34 /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid
root    34373     1 34373 34373    0 Is    ??    0:00.00 /usr/local/bin/rsync --daemon
root    34398     1 34398 34398    0 Is    ??    0:00.00 ntpd: [priv] (ntpd)
_ntp    34399 34398 34398 34398    0 S     ??    0:00.18 ntpd: ntp engine (ntpd)
_ntp    34400 34399 34398 34398    0 S     ??    0:00.28 ntpd: dns engine (ntpd)
vscan   34471     1 34471 34471    0 Ss    ??    0:07.68 /usr/local/sbin/amavisd (master) (perl)
vscan   34476 34471 34471 34471    0 I     ??    0:00.01 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34477 34471 34471 34471    0 I     ??    0:00.01 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34478 34471 34471 34471    0 I     ??    0:00.01 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34479 34471 34471 34471    0 I     ??    0:00.01 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34480 34471 34471 34471    0 I     ??    0:00.01 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34481 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34482 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34483 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34484 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34485 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34486 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34487 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34488 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34489 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34490 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
vscan   34491 34471 34471 34471    0 I     ??    0:00.02 /usr/local/sbin/amavisd (virgin child) (perl)
mysql   34539     1 34539 34539    0 Is    ??    0:00.04 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadi
mysql   34953 34539 34539 34539    0 S     ??   11:45.69 [mysqld]
root    35043     1 35043 35043    0 Ss    ??    0:07.58 /usr/local/sbin/httpd
root    35429     1 35429 35429    0 Is    ??    0:00.47 /usr/local/libexec/postfix/master
postfix 35431 35429 35429 35429    0 I     ??    0:00.65 qmgr -l -t fifo -u
postfix 38023 35429 35429 35429    0 I     ??    0:00.04 pickup -l -t fifo -u
www     39164 35043 35043 35043    0 R     ??    1:37.76 /usr/local/sbin/httpd
www     39225 35043 35043 35043    0 R     ??    0:16.31 /usr/local/sbin/httpd
www     39229 35043 35043 35043    0 I     ??    0:11.93 /usr/local/sbin/httpd
www     39230 35043 35043 35043    0 I     ??    0:06.73 /usr/local/sbin/httpd
www     39231 35043 35043 35043    0 I     ??    0:08.59 /usr/local/sbin/httpd
www     39241 35043 35043 35043    0 I     ??    0:06.39 /usr/local/sbin/httpd
www     39242 35043 35043 35043    0 I     ??    0:06.99 /usr/local/sbin/httpd
www     39244 35043 35043 35043    0 R     ??    0:56.06 /usr/local/sbin/httpd
www     39257 35043 35043 35043    0 S     ??    0:03.21 /usr/local/sbin/httpd
www     39259 35043 35043 35043    0 S     ??    0:00.12 /usr/local/sbin/httpd
www     39264 35043 35043 35043    0 R     ??    0:02.95 /usr/local/sbin/httpd
www     39265 35043 35043 35043    0 S     ??    0:03.23 /usr/local/sbin/httpd
www     39271 35043 35043 35043    0 I     ??    0:08.46 /usr/local/sbin/httpd
www     39277 35043 35043 35043    0 S     ??    0:02.98 /usr/local/sbin/httpd
www     39289 35043 35043 35043    0 I     ??    0:02.55 /usr/local/sbin/httpd
www     39290 35043 35043 35043    0 S     ??    0:04.95 /usr/local/sbin/httpd
www     39292 35043 35043 35043    0 S     ??    0:03.29 /usr/local/sbin/httpd
www     39302 35043 35043 35043    0 R     ??    0:15.48 /usr/local/sbin/httpd
www     39303 35043 35043 35043    0 I     ??    0:02.97 /usr/local/sbin/httpd
www     39309 35043 35043 35043    0 I     ??    0:00.05 /usr/local/sbin/httpd
www     39311 35043 35043 35043    0 I     ??    0:00.50 /usr/local/sbin/httpd
www     39317 35043 35043 35043    0 I     ??    0:00.01 /usr/local/sbin/httpd
www     39318 35043 35043 35043    0 S     ??    0:03.35 /usr/local/sbin/httpd
www     39320 35043 35043 35043    0 I     ??    0:02.20 /usr/local/sbin/httpd
www     39322 35043 35043 35043    0 I     ??    0:04.64 /usr/local/sbin/httpd
www     39325 35043 35043 35043    0 S     ??    0:00.00 /usr/local/sbin/httpd
www     39326 35043 35043 35043    0 S     ??    0:00.00 /usr/local/sbin/httpd
www     39327 35043 35043 35043    0 S     ??    0:00.00 /usr/local/sbin/httpd
root    66376  1531 66376 66376    0 Ss    ??    0:23.63 sshd: duce@pts/0 (sshd)
root    66384  1531 66384 66384    0 Is    ??    0:00.70 sshd: duce@pts/2 (sshd)
root    66392  1531 66392 66392    0 Is    ??    0:01.06 sshd: duce@pts/7 (sshd)
root    85108     1 85108 85108    0 Is    ??    0:00.63 /usr/sbin/cron -s
root    98585  1531 98585 98585    0 Is    ??    0:00.78 sshd: duce@notty (sshd)
duce    98586 98585 98586 98586    0 Is    ??    0:00.03 -zsh (zsh)
root     1758     1  1758  1758    0 Is+   v0    0:00.00 /usr/libexec/getty Pc ttyv0
root     1759     1  1759  1759    0 Is+   v1    0:00.00 /usr/libexec/getty Pc ttyv1
root     1760     1  1760  1760    0 Is+   v2    0:00.00 /usr/libexec/getty Pc ttyv2
root     1761     1  1761  1761    0 Is+   v3    0:00.00 /usr/libexec/getty Pc ttyv3
root     1762     1  1762  1762    0 Is+   v4    0:00.00 /usr/libexec/getty Pc ttyv4
root     1763     1  1763  1763    0 Is+   v5    0:00.00 /usr/libexec/getty Pc ttyv5
root     1764     1  1764  1764    0 Is+   v6    0:00.00 /usr/libexec/getty Pc ttyv6
root     1765     1  1765  1765    0 Is+   v7    0:00.00 /usr/libexec/getty Pc ttyv7
root    39328 66402 39328 66377    2 R+     0    0:00.00 ps -axj
root    39329 66402 39328 66377    2 DL+    0    0:00.00 more
duce    66377 66376 66377 66377    0 IWs    0    0:00.00 -zsh (zsh)
root    66401 66377 66401 66377    1 IW     0    0:00.00 su
root    66402 66401 66402 66377    1 S      0    0:00.22 su (zsh)
duce    31944 31942 31944 31944    0 Is+    1    0:00.22 zsh
root    39128 66406 39128 66385    1 I+     2    0:00.00 tail -f /var/log/maillog
duce    66385 66384 66385 66385    0 IWs    2    0:00.00 -zsh (zsh)
root    66405 66385 66405 66385    1 IW     2    0:00.00 su
root    66406 66405 66406 66385    1 I      2    0:31.67 su (zsh)
duce    31945 31942 31945 31945    0 IWs    3    0:00.00 zsh
root    31980 31945 31980 31945    1 IW     3    0:00.00 su
root    31982 31980 31982 31945    1 I+     3    0:02.79 su (zsh)
duce    31946 31942 31946 31946    0 IWs    4    0:00.00 zsh
root    31987 31946 31987 31946    1 IW     4    0:00.00 su
root    31988 31987 31988 31946    1 I+     4    0:03.82 su (zsh)
duce    31965 31942 31965 31965    0 IWs    5    0:00.00 zsh
root    31991 31965 31991 31965    1 IW     5    0:00.00 su
root    31993 31991 31993 31965    1 I+     5    0:02.01 su (zsh)
root    14417 69150 14417 31972    1 S+     6    0:02.67 tail -f /var/log/apache/DOMAIN-access.log
duce    31972 31942 31972 31972    0 IWs    6    0:00.00 zsh
root    69149 31972 69149 31972    1 IW     6    0:00.00 su
root    69150 69149 69150 31972    1 IW     6    0:00.00 su (zsh)
root    34415     1 34415 66393    0 I      7    0:00.01 /usr/local/sbin/courierlogger -pid=/var/run/imapd.pid -start -name=imapd /usr/local/libexec/courier
root    34416 34415 34416 66393    1 I      7    0:00.01 /usr/local/libexec/courier-imap/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noide
root    34430     1 34430 66393    1 I      7    0:00.01 /usr/local/sbin/courierlogger -pid=/var/run/pop3d.pid -start -name=pop3d /usr/local/libexec/courier
root    34431 34430 34431 66393    1 I      7    0:00.01 /usr/local/libexec/courier-imap/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noide
root    34452     1 34452 66393    0 I      7    0:00.01 /usr/local/sbin/courierlogger -facility=mail -pid=/var/run/authdaemond/pid -start /usr/local/libexe
root    34453 34452 34452 66393    0 S      7    0:00.01 /usr/local/libexec/courier-authlib/authdaemond
root    34464 34453 34452 66393    0 S      7    0:00.00 /usr/local/libexec/courier-authlib/authdaemond
duce    66393 66392 66393 66393    0 IWs    7    0:00.00 -zsh (zsh)
root    66409 66393 66409 66393    1 IW     7    0:00.00 su
root    66410 66409 66410 66393    1 I+     7    0:07.85 su (zsh)
duce     1675  1673  1675  1675    0 IWs    8    0:00.00 -zsh (zsh)
duce     1688  1675  1688  1675    1 S+     8    0:00.24 screen -x
duce    69128 31942 69128 69128    0 IWs    9    0:00.00 zsh
root    69162 69128 69162 69128    1 IW     9    0:00.00 su
root    69165 69162 69165 69128    1 I+     9    0:00.55 su (zsh)
root    16372 69130 16372 69130    1 IW    10    0:00.00 su
root    16373 16372 16373 69130    1 I+    10    0:13.64 su (zsh)
duce    69130 31942 69130 69130    0 IWs   10    0:00.00 zsh
root    19841 85521 19841 85521    1 IW    11    0:00.00 su
root    19843 19841 19843 85521    1 IW    11    0:00.00 su (zsh)
root    19846 19843 19846 85521    1 D+    11  309:35.46 clamscan -ir /usr/local/www
duce    85521 31942 85521 85521    0 IWs   11    0:00.00 zsh
duce     3216  3214  3216  3216    0 IWs   12    0:00.00 -zsh (zsh)
root    16367  3216 16367  3216    1 IW    12    0:00.00 su
root    16368 16367 16368  3216    1 IW    12    0:00.00 su (zsh)
root    16582 16368 16582  3216    1 IW+   12    0:00.00 man clamscan
root    16583 16582 16582  3216    1 IW+   12    0:00.00 sh -c /usr/bin/zcat /usr/local/man/cat1/clamscan.1.gz | more -s
root    16585 16583 16582  3216    1 I+    12    0:00.03 more -s

I actually had clam and amavis disabled as I thought that might be the culprit as it is not a mail server. Only the webserver sends emails.

 

[obsigna]

Looking at the TIME column of the output of the ps command, I can identify only clamscan as the major load of the CPU. Nonetheless 309:35 does account for a total load of 11 %. Depending on the number of CPU cores, top(1) would show a higher percentage, because its calculation basis is a single core. So, on a 4 core machine 11 % would show 44 % in top. There is also some higher load caused by interrupt handling.

I am not an expert in clam and amavis, however, antivirus scanner software in general is known to impose high CPU loads. In addition, it does not seem very useful to scan the whole /usr/local/www directory tree. The majority of the content of the webroot is supposed to be 100 % under control of the web administrator, so as long as you can trust yourself (say your admin), you don't need to AV scan that directory as a whole, only content which may be uploaded by users.

 

[duce]

obsigna - CPU is AMD Athlon(tm) II X2 265 Processor (3315.10-MHz K8-class CPU).
Scan finished and found nothing. Have disabled clam and amavis again.
And yes I'm quite vigilant webserver side specially with file and folder permissions. Mostly run Wordpress websites which I constantly update all at once together with plugins via subversion from CLI. The script I run even sets file and folder permissions just in case I forgot something working on a website during updating.

I do suspect it is an interrupt that is an issue somewhere or kernel incorrectly configured by me.

 

[asteriskRoss]

Could I ask that someone feeling like doing some freelance work contact me please.

duce, if you are still looking for someone you may attract more interest by sending a message to the freebsd-jobs mailing list.

 

[SirDice]

Mostly run Wordpress websites which I constantly update all at once together with plugins via subversion from CLI.

This is a common entry point for malware. Especially the plugins, even if you keep everything up to date.

Scanning typically doesn't find anything. This type of malware is usually quite customized and isn't recognized by malware scanners. If I write an exploit for a website and release it into the wild your scanner simply won't pick it up until it's found and analyzed. Once it's found it may be detected but if I make just a few changes the detection is easily thwarted. In short, never rely on malware scanners. They're only as good as their signatures.

 

[gkontos]

This is a common entry point for malware. Especially the plugins, even if you keep everything up to date.

Scanning typically doesn't find anything. This type of malware is usually quite customized and isn't recognized by malware scanners. If I write an exploit for a website and release it into the wild your scanner simply won't pick it up until it's found and analyzed. Once it's found it may be detected but if I make just a few changes the detection is easily thwarted. In short, never rely on malware scanners. They're only as good as their signatures.

Quite right. Especially clamav in my experience is really useless for that part. But a hacked wordpress site should produce some traffic that you can easily sniff.

 

[SirDice]

But a hacked wordpress site should produce some traffic that you can easily sniff.

Definitely. It's common to find some backdoor PHP script in a place you don't expect. They then use this backdoor to "proxy" spam or DDoS attacks. Check the various directories if you allow uploads (pictures for example). These backdoors are typically uploaded by exploiting bugs in plugins and their names look like regular pictures. They are however PHP scripts. I've also seen malware that injects itself into regular Wordpress pages. Unless you know what to look for they're easily overlooked.

 

[duce]

Fully agree, thanks guys. But on the one hand I do not use FTP nor allow the upload of anything but jpeg, png, gif and pdf. Especially PHP is blocked. Anything else has to be uploaded by myself using SCP. But nevertheless no matter how vigilant you are there is always the unknown element of what a user/client would/could do behind your back. And then there is the possibility of using a plugin that has been compromised. Will check in more detail.
For now I got the loads down to below 10's by firewalling some IP ranges I see in the webserver logs acting like bots. That alone tells me there just might be validity to such a compromise.
Thanks for the advice.