For SMTP i'm using exim (switched from postfix because it was unable to suit my needs of mail processing), and for POP3 / IMAP i'm using Сyrus-Imapd. But it all depends on what mail system you're trying to build. How many users (and then - what type of DB for authentication - SQL/LDAP?)? Preferred SPAM fighting software? Webmail frontend? Specific mail processing (i.e. rewriting addresses, saving all mail, etc)?