Recommendation on Spam Filter

Since I moved our mail server to a new machine, my spam has increased 4000%. That is not an exaggerated number either. I have gone from around 25 or so per day to well over a 1,000.

Plesk was handling the spam rules so I never really paid attention to them. Plesk is gone, never to return so now I need to come up with a method to control this.

Any suggestions on what package I should consider using? All thoughts and comments would be greatly appreciated.

Douglas
 
I prefer spam blocking to spam filtering - it blocks the spam from being accepted rather than accepting everything and then trying to weed through it. It does this by targeting techniques used by spammers rather than trying to analyze each message to determine whether it's spam or not.

Greylisting is highly effective with 0 chance of false positives. It simply delays mail from reaching it's destination for n minutes, simulating a temporary failure. All standards-compliant mail servers MUST retry, so it works well with little disruption. Yeah, it's frustrating waiting 5 minutes to get a particular message from a new sender when you're on the phone with the person, but the amount of spam it blocks is unbelievable.

Another good method is to use one or more RBLs such as spamhaus, spamcop, etc to block senders based on sender address. RBLs list addresses known to be sources of spam.

There are also various restrictions that you can employ on your mail server to make it much more difficult for spam to get in. A few of these restrictions that I have set up on my mail server include:

Require a HELO statement from the client.

Reject any HELO commands from a computer that identifies itself as my server. (Since no other machine is me, so it works well.)

Require a fully qualified domain name in the HELO.

Reject invalid host names in the HELO.

Reject mail from senders with an invalid domain.

Reject mail from machines who have gotten spam through my blocks. (The first one in the ~3 years my mail server has been live was added yesterday.)


All of these techniques block spam before it's accepted at a very low cost of CPU time. I also have amavisd-new set up with clamav to filter out virus-laden mail. I do not have any true spam filter installed and the instances of real spam for my users is very low. (By 'real spam' I mean truly unsolicited mail. Some stupid people sign up for mailing lists and then complain when they get mail from the list, believing it to be spam.)

You don't mention what MTA you're using. If it's Postfix, I'd be willing to post snippets of my config for you. I'd highly suggest reading up on them before using them on your box, but they've been great for me.
 
I am using sendmail with qpopper as the pop server. In addition I have dovecot loaded for use with squirrelmail.
 
Properly configured Postfix plus policyd-weight (default config) will drop your spam from 1000's to dozens or less.

Tweak policyd-weight and you might see a few spam a week.

Policyd-weight blocks before crap enters your mail queue. I prefer this over greylisting because it doesn't impede the flow of legitimate mail at all.

Note I haven't talked about content inspection at all... not necessary really, but I do have a bayesian inspection solution which is used for tagging only - all marked mail gets delivered albeit put in a junk folder. I use something simple for this.
 
A list of methods I'm currently using. It's quite similar to the one posted above.

First comes the spamd blacklists (uatraps and nixspam).

Then I check the email sender and recipients, as well as the helo hostnames. Next, SPF records are looked up, DNSBL blacklists are queried. Greylisting is implemented afterwards. Last of all, there's sender verification (a Postfix feature).

All of the methods aren't CPU intensive. They just require a bit of time and bandwidth.

After that, SpamAssassin/bogofilter is used to classify the received mail as spam or ham. This is the part that's CPU intensive. SpamAssassin has default rules, but it requires a bit of training and tuning. Bogofilter doesn't have rules like SpamAssassin; it uses wordlists to classify spam and ham. Bogofilter is extremely accurate after it's trained. Both SpamAssassin use Bayesian filtering, but SpamAssassin is more CPU intensive than bogofilter.

When using SpamAssassin, I use a plugin to integrate ClamAV. There's also FuzzyOCR. Right now I haven't found a way to do this with bogofilter.
 
mwatkins said:
Properly configured Postfix plus policyd-weight (default config) will drop your spam from 1000's to dozens or less.

Tweak policyd-weight and you might see a few spam a week.

Policyd-weight blocks before crap enters your mail queue. I prefer this over greylisting because it doesn't impede the flow of legitimate mail at all.

Note I haven't talked about content inspection at all... not necessary really, but I do have a bayesian inspection solution which is used for tagging only - all marked mail gets delivered albeit put in a junk folder. I use something simple for this.

Do you have any documentation sources for policyd-weight?
I'm currently using Postfix + Dovecot (Spam is as you would expect rife, on a single address I receive over 30 per day).

EDIT:
Does anyone have any experience with "postfwd"? Given the maintenance has effectively stopped on policyd-weight.
 
Before I found FreeBSD and was running Linux, I had a very aggressive spam filter running in conjunction with the Exim mail server -- however it was barely a notch above Sendmail (I never felt comfortable with Exim -- the config files are difficult to understand).. The brunt of the work was done by SA-Exim which was a module that did spam checking in real-time and would teergrube (nifty term they coined) any spam senders which basically would hold the connection open until they dropped it on their end -- effectively delaying that sender from delivering more spam to others. A nifty concept that worked pretty well but was difficult to maintain w/ Exim.. When I moved over to FreeBSD, I dumped Exim in favor of .. ugg.. Sendmail as the how-to's were abundant and easy to comprehend. When used with SpamAssassin, ClamAV,greylisting, it worked pretty well but still let through quite a bit of spam daily and occasionally kicked out legitimate emails.. I like some of the ideas presented earlier in this thread and will try to get them set up once I get my Postfix working after moving to FreeBSD 8 this past weekend..
 
hiatek said:
Do you have any documentation sources for policyd-weight?

While the original author has indicated he is not keen to support policyd forever, I do continue to use policyd-weight because it's a simple solution that works. It would not surprise me to see the community continue to support it (perhaps they already are), but if that proves not to be the case and I don't want to self-maintain the perl code, I might rewrite it in Python or adopt another solution. For now it is so reliable that I see no need to adopt anything else, and installation and configuration is so easy I'm too lazy to hunt down another solution.

Documentation - the HOWTO is alright; you might prefer to avail yourself of the port and install that. Essentially its a single perl script and a config file, plus two modifications to Postfix, one in main.cf, here's a stripped down example that would probably form a good start for you:

Code:
smtpd_recipient_restrictions =
   permit_mynetworks,
   # place non-recipient restrictions AFTER the reject_unauth_destination restriction
   reject_unauth_destination,
   # - this trims a ton of spam; monitor to see if anay legit senders are blocked
   reject_non_fqdn_sender,
   reject_non_fqdn_recipient,
   reject_non_fqdn_hostname,
   reject_invalid_hostname,
   reject_unknown_sender_domain,
   # me not final dest; RCPT TO domain issues
   reject_unknown_recipient_domain,
   # policyd_weight - last thing before "permit"
   check_policy_service inet:127.0.0.1:12525,
   permit

# POLICYD-WEIGHT SETTINGS
# see also: http://www.policyd-weight.org/
smtpd_policy_service_max_idle = 1800s
smtpd_policy_service_max_ttl = 3600s
policy_time_limit = 3730



And in master.cf, inserted in the section near "maildrop":

Code:
# ...
# policyd-weight - starts it up for us
policy  unix  -       n       n       -       -       spawn
  user=polw argv=/usr/local/bin/policyd-weight
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
#maildrop  unix  -       n       n       -       -       pipe
#  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
#
# ...

Start with the default config file and see how that works for you. For certain you'll see your incoming spam load cut in half, probably by quite a bit more. If you like I can send you my modified config file which includes a GeoIP section (there's a patch out there to include GeoIP support for policyd-weight) and other customizations. I also have modified policyd itself slightly but doubt you need to go there at this time.

For content filtering for a relatively small (dozens) group of users you might try using bogofilter. I've never used it for a larger user population but for a relatively small group it actually seems ideal - once trained it's a set and forget solution, doesn't seem to break, ever, and presents a fairly low load on the server which is not true of many of the common content filtering solutions out there.

I've got bogofilter configured to simply tag what little spam makes it through the postfix - policyd-weight gauntlet. It really is a small percentage that remains, much of it Nigerian spam style stuff. These are delivered to a SPAM folder for local IMAP users, pop users get to see it regardless but their confidence level that SPAM is indeed spam I'm told is very high.
 
mwatkins,
I have installed policyd-weight from ports collection and followed your advice in setting it up.

I created the section in main.cf for smtpd_recipient_restrictions and policyd-weight section.
I couldn't find a section that I could uncomment.
I also added the lines in master.cf.

Do I need to create a file for policyd-weight or does it just work with a default if none are created?

I see the sample file is there, although I havent renamed or done anything with it.

Also, do I need to set it to start somehow on boot?


Regards,
Michael
 
If I recall correctly, you can use the defaults if you wish..

To get it to spawn at startup just put the typical entry in your /etc/rc.conf file (e.g. policyd_weight_enable="YES")

I'm just toying around with it -- hoping to get rid of about 10+ more spam per day:e
 
By the way.. It seems like Postfix allows using multiple policy servers in the same 'set' as shown below:

Code:
# the following checks apply right after the client sends the RCPT: 'to' command
smtpd_recipient_restrictions = 
              reject_invalid_hostname,  
              reject_non_fqdn_hostname,  
              reject_non_fqdn_sender,  
              reject_non_fqdn_recipient,  
              reject_unknown_sender_domain,  
              reject_unknown_recipient_domain,  
              permit_mynetworks,  
              reject_unauth_destination,  
              check_recipient_access hash:$config_directory/access_recipient,
              check_helo_access hash:$config_directory/helo_checks,
              check_sender_access hash:$config_directory/access_sender,
              check_client_access hash:$config_directory/client_access,
              reject_rbl_client zen.spamhaus.org,
[color="Navy"][B]              check_policy_service inet:127.0.0.1:10023,
              check_policy_service inet:127.0.0.1:12525,[/B][/color]
              permit

Anyone see any issues with doing this? See some output below for reference -- notice both Postgrey and PolicyD running concurrently.. In this case PolicyD appears to append some stuff but Postgrey trumps it with greylisting :

Code:
Mar 22 23:26:08 smtp postfix/master[64059]: daemon started -- version 2.7.0, configuration /usr/local/etc/postfix
Mar 22 23:29:19 smtp postfix/smtpd[64074]: connect from trcont.eu[82.209.198.117]
Mar 22 23:29:21 smtp postgrey[1624]: action=greylist, reason=new, client_name=trcont.eu, client_address=82.209.198.117, sender=b.ashford@ymail.com, recipient=me@mydomain.com
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: child: spawned 
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: cache_query: ask 82.209.198.117  b.ashford@ymail.com ymail.com 
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: cache_query: "ask82.209.198.117b.ashford@ymail.com 0" vs "ask82.209.198.117b.ashford@ymail.com " 
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: 117.198.209.82.pbl.spamhaus.org, 21904 
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: rbl_lookup: 117.198.209.82.pbl.spamhaus.org vs 117.198.209.82.pbl.spamhaus.org, 21904 vs 21904, anc == 0 
Mar 22 23:29:21 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: 117.198.209.82.sbl-xbl.spamhaus.org, 49676 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: 117.198.209.82.sbl-xbl.spamhaus.org vs 117.198.209.82.sbl-xbl.spamhaus.org, 49676 vs 49676, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: 117.198.209.82.bl.spamcop.net, 62652 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: 117.198.209.82.bl.spamcop.net vs 117.198.209.82.bl.spamcop.net, 62652 vs 62652, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: 117.198.209.82.dnsbl.njabl.org, 29799 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: 117.198.209.82.dnsbl.njabl.org vs 117.198.209.82.dnsbl.njabl.org, 29799 vs 29799, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: 117.198.209.82.ix.dnsbl.manitu.net, 17951 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: 117.198.209.82.ix.dnsbl.manitu.net vs 117.198.209.82.ix.dnsbl.manitu.net, 17951 vs 17951, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: ymail.com.multi.surbl.org, 14081 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: ymail.com.multi.surbl.org vs ymail.com.multi.surbl.org, 14081 vs 14081, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: ymail.com.rhsbl.ahbl.org, 43366 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: ymail.com.rhsbl.ahbl.org vs ymail.com.rhsbl.ahbl.org, 43366 vs 43366, anc == 0 
Mar 22 23:29:22 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: ymail.com.dsn.rfc-ignorant.org, 37681 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: rbl_lookup: ymail.com.dsn.rfc-ignorant.org vs ymail.com.dsn.rfc-ignorant.org, 37681 vs 37681, anc == 0 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: ymail.com.postmaster.rfc-ignorant.org, 14665 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: rbl_lookup: ymail.com.postmaster.rfc-ignorant.org vs ymail.com.postmaster.rfc-ignorant.org, 14665 vs 14665,  anc == 1 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: rbl_lookup: sending: ymail.com.abuse.rfc-ignorant.org, 37202 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: rbl_lookup: ymail.com.abuse.rfc-ignorant.org vs ymail.com.abuse.rfc-ignorant.org, 37202 vs 37202,  anc == 1 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: weighted check:  NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_EQ_HELO_MX=-3.1 (check from: .ymail. - helo: .trcont. - helo-domain: .trcont.)  FROM/MX_MATCHES_NOT_HELO(DOMAIN)=1 IN_PM_RFCI=3.2 IN_ABUSE_RFCI=3.2 <helo_ips:  trcont.eu 98.137.54.238 66.94.236.34 67.195.168.31 74.6.136.65 206.190.54.127 209.191.88.254 67.195.168.230 98.137.54.237 68.180.206.184 206.190.60.37 82.209.198.117>; <instance=fa4a.4ba85fc1.1badd.082.209.198.117b.ashford@ymail.com> <client=82.209.198.117> <helo=trcont.eu> <from=b.ashford@ymail.com> <to=me@mydomain.com>; rate: -0.2 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: cache_query: padd 82.209.198.117 -0.2 b.ashford@ymail.com ymail.com 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: cache_query: "padd82.209.198.117b.ashford@ymail.com 0" vs "padd82.209.198.117b.ashford@ymail.com " 
Mar 22 23:29:23 smtp postfix/policyd-weight[64082]: decided action=PREPEND X-policyd-weight:  NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_EQ_HELO_MX=-3.1 (check from: .ymail. - helo: .trcont. - helo-domain: .trcont.)  FROM/MX_MATCHES_NOT_HELO(DOMAIN)=1 IN_PM_RFCI=3.2 IN_ABUSE_RFCI=3.2 <helo_ips:  trcont.eu 98.137.54.238 66.94.236.34 67.195.168.31 74.6.136.65 206.190.54.127 209.191.88.254 67.195.168.230 98.137.54.237 68.180.206.184 206.190.60.37 82.209.198.117>; rate: -0.2; <instance=fa4a.4ba85fc1.1badd.0> <client=82.209.198.117> <helo=trcont.eu> <from=b.ashford@ymail.com> <to=me@mydomain.com>; delay: 2s 
Mar 22 23:29:23 smtp postfix/smtpd[64074]: NOQUEUE: reject: RCPT from trcont.eu[82.209.198.117]: 450 4.2.0 <me@mydomain.com>: Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/mydomain.com.html; from=<b.ashford@ymail.com> to=<me@mydomain.com> proto=ESMTP helo=<trcont.eu>
Mar 22 23:29:23 smtp postfix/smtpd[64074]: disconnect from trcont.eu[82.209.198.117]
 
osx-addict said:
If I recall correctly, you can use the defaults if you wish..

To get it to spawn at startup just put the typical entry in your /etc/rc.conf file (e.g. policyd_weight_enable="YES")

I'm just toying around with it -- hoping to get rid of about 10+ more spam per day:e

Okay,
So I have it working and its only letting valid email through, compared to 50+ a day before policd-weight.

Problem being getting it started on boot, if I dont start it, I dont seem to get e-mail at all.

I have tried your suggestion of:
Code:
policyd_weight_enable="YES"

This does not seem to start it though when i reboot the server.
I have tried [cmd=]ps aux |grep policyd[/cmd].
I get no results, which leads me to believe it is not starting.
 
hiatek said:
Okay,
This does not seem to start it though when i reboot the server.
I have tried [cmd=]ps aux |grep policyd[/cmd].
I get no results, which leads me to believe it is not starting.

What happens if you issue "/usr/local/etc/rc.d/policyd-wieght start" ? Does it complain?
 
As mentioned, I can start it manually. I require it to start on boot. I.E. if I shut the machine down for maintenance or extreme power failure and the UPS runs out.
It would be a pain to have to do anything other than push the power button, especially if I'm out for the day but I can get someone push a button for me, compared to giving them roots password and trying to tell them how to SSH and then login and issue a command.

Also, I'm not sure why my post was edited by dutchdaemon?


Kind Regards,
Michael
 
hiatek said:
As mentioned, I can start it manually. I require it to start on boot. I.E. if I shut the machine down for maintenance or extreme power failure and the UPS runs out.
It would be a pain to have to do anything other than push the power button, especially if I'm out for the day but I can get someone push a button for me, compared to giving them roots password and trying to tell them how to SSH and then login and issue a command.

Look in the startup script - should be in /usr/local/etc/rc.d/ - and look at what parameter it's expecting in /etc/rc.conf. Then add <whatever>_enable="YES" to /etc/rc.conf and reboot. If that does not work, look in /var/log/messages and see if it's starting and erroring out, not starting, etc.


hiatek said:
Also, I'm not sure why my post was edited by dutchdaemon?

I don't know for sure, but you probably didn't put in either a code or cmd tag to format the information so it displays well.
 
Thanks DutchDaemon for editing.

Thanks Ruler for giving me that advice, it made a light switch on for me (not literally, more metaphoric).
If I follow correctly, when I have a similar issue with different software, I can check this directory and the file pertaining to the software?

The policyd-weight now starts correctly on boot. I am a happy man.


Regards,
Michael
 
hiatek said:
Thanks Ruler for giving me that advice, it made a light switch on for me (not literally, more metaphoric).
If I follow correctly, when I have a similar issue with different software, I can check this directory and the file pertaining to the software?

If any post on here literally makes a light turn on, that would be impressive! ;)

Your assumption is correct. The files in /etc/rc.d and /usr/local/etc/rc.d are simply scripts to start/stop/restart various daemons. You can view any of them to figure out what they're doing and what they expect to be where.
 
Hi.

I'm working in my first spam server, I want to have a dedicated machine for this purpose and if this machine accept the email, will be forward to each user mailbox in other machine(email server).

I want to understand something, I setup postfix-policyd-weight, this will be my first wall for spammers.

This spam machine will have:

postfix-policyd-weight
amavisd-new
clamd
spamassassin

Once the inbound email pass the first filter in this case policyd-weight, latter will return to postfix and see if we have other settings before sending the email to the user mailbox, example:

Code:
smtpd_recipient_restrictions =
        permit_mynetworks,
        reject_unauth_destination,
        reject_non_fqdn_sender,
        reject_non_fqdn_recipient,
        reject_non_fqdn_hostname,
        reject_invalid_hostname,
        reject_unknown_sender_domain,
        reject_unknown_recipient_domain,
        check_policy_service inet:127.0.0.2:12525
        permit

#Amavis Injection
content_filter = smtp-amavis:[127.0.0.2]:10024

1; The email will reach the policyd-weight.
2; If pass step 1, the email will reach amavisd-new
This one will call clamd, spamassassin
3; If pass, the email will be forward to the mailbox in the email server.

Do I'm right?

The email will follow postfix settings from the top to the botton?

I have to setup a rule in my firewall to rdr communication in to my wan interface to port 25 to port 12525.

Thanks all for this interesting post.
 
Don't forget grey listing via something like post-grey.. I have that before policyd-weight.. Mine is tailored after this installation taken from the Postfix Howto's and FAQs page. The big thing to note is that order is very important.. I'm still tweaking mine but do not yet have SpamAssassin running..
 
Personally I do not like grey listing; it might be ok for a limited number of users but for a diverse or large group, it is painful to manage. Some senders just don't behave correctly when asked to retry, and I've even seen some telco's running ISPs fall into that category.
 
It's quite true that greylisting does cause problems since some mail servers don't retry.

However, this shouldn't be a problem since Postgrey keeps a whitelist of servers that have problems.
 
I've used greylisting for years on my company's mail server and haven't had a problem. The only mail servers that should have problems are those that do no adhere to standards because temporary delivery deferrals are part of the RFCs regarding mail servers. (Think if an internet line or server goes down temporarily - has the same effect as greylisting, simply saying 'this message cannot be delivered now - try later' to the server sending the message.)

Also, it's unbelievable how much spam it kills. A now ex-employee thought it'd be funny to sign up a co-worker for lots of spam lists; he was getting 300+ spam messages every day. I changed the target's address after the jerk was fired and removed the old address. With 200 messages that are deferred for his old address via postgrey, there are maybe 15-20 user not found errors in the logs for his old account. While not a scientific test, it is pretty impressive IMO.
 
Back
Top