I often use this:
https://mxtoolbox.com/diagnostic.aspx
They run a whole slew of tests on your mailserver to see if it can be abused to relay. But it's obviously only useful for mailservers attached to the internet.
Hm, it thinks spamd is an open relay
I usually use
https://ssl-tools.net/mailservers and ssllabs.com for webservers. But using any test tool at all is already a good start. But you shouldn't only test for encrypted channels, also test if a service is _only_ available via SSL/TLS and doesn't accept plaintext connections. Except of course the incoming/outgoing SMTP in case some old/weird/crappy server still doesn't support it but you need to get mails delivered from/to that server... (yes, there are still plenty of them out there
)
@Topic:
As others have pointed out: make yourself familiar with HOW mails are sent around the internet (SMTP). Sendmail might be a hard entry point due to its arcane config syntax.
Postfix is nicely modular and can run with a very minimal config; also it is very well documented and due to its very wide usage (IIRC >60% market share) there are tons of good books and "quickstart" guides as well as solutions for very specific problems out there.
For a quick, simple and secure private mail server I'd suggest looking at OpenSMTPd. It's very clean and configuration is extremely easy to understand - if you understand PF rulesets, you will understand the OpenSMTPd config syntax; they're very similar. Also the defaults are very sane and secure - in contrast to other MTAs, e.g. postfix defaulted a long time to "free for all", hence there were tons of open relays in the wild...
I've been running postfix/dovecot based mailservers for >10 years now, so here are some general suggestions that come to mind:
- Pick your MTA (and other parts) depending on your needs, not how much (unused) features it has, and how comfortable you are with its layout/architecture and configuration syntax. E.g. getting your head around how the delivery chain in postfix works is absolutely essential to later add filters/milters and inject/extract mail at the right point in the chain.
Same goes for the MDA and other parts of the mailserver (e.g. database for users if needed; webmail; spam filter etc...). It makes no sense to use the software with most bells & whistles if you don't need them and are overwhelmed with its configuration. In fact it is only dangerous and often frustrating.
Even if its quite a heavyweight, dovecot would also be my suggestion for IMAP - it is very powerful but can run with a very minimal configuration and subset of its features enabled. The default configuration shipped with FreeBSD is very well documented and essentially "everything switched off" - so you have to specifically enable everything first, which often boils down to uncomment and adjust a few lines in the example config.
As an addition to the official documentation I can highly recommend the dovecot book from Peer Heinlein. The author has been running big mailserver installations for >20 years, so the book (as well as his Postfix book, but AFAIK it's only available in german) is full of real-world examples and very helpful suggestions/recommendations.
- Add SSL to _all_ externally available services at an early stage, so you won't have to change stuff around everywhere at the end. Use the staging environment of let's encrypt during testing/setup; there's nothing more annoying than running into the rate-limiting in the middle of configuration/testing...
- Always use the tightest access rules possible; never "give up" and let everything pass just to get something working; this WILL bite you in the (usually very near) future! If your domain ends up on a spamlist its _very_ hard to use it reliably for email for quite a long time and even months later you will occasionally get delivery failures.
- Put every service in its own jail; only run PF and spamd on the host. I've been setting up my last 2 mail servers like this and it might have been one of the best choices I've made in my life during the last 3 years (well, not necessarily because my life choices are usually stupid...
). Not only confines this everything in case of a security flaw (be it a bug or misconfiguration), but it also makes maintenance and scaling a walk in the park.
- Don't have A good backup plan - have several of them! Never only rely on snapshots/fullbackups of a VPS by your provider. They usually break exactly the week before you need them. Put the mailstore on ZFS; this makes it easy to send snapshots to a backup storage as well as an off-site pool (e.g. a hot-standby machine) and/or a backup host that dumps them on tape. If the server handles anything remotely connected to business emails, have 2 backup paths more than you think you need - you will need them! (It also makes scaling very easy - just add more storage or attach more/bigger block storage to the VPS...)
- Don't let yorself talk into using whitelists to bypass security checks e.g. for misbehaving/outdated mail clients or stupidly written contact forms on websites. These are a constant source of trouble, tears and screaming and _always_ lead to a major incident one day. Best Answer to such requests: "technically not feasable". Whoever thinks he's technically adept to challenge that statement gets an additional: "due to a customized setup to fulfill our business needs [...] only with major changes which will impact availability [...] less features afterwards [...] need you to sign all these points off by 'my/our/yor boss' to make the requested changes....". I have long resigned to discuss the technical details and security impacts esp. with 'web designers' - just give them the path of highest resistance as the only way to go forward.
