Do you have a simple home network? At least something you can put a few machines on. That's probably enough to play with. Just make sure the mail server can't connect to the outside world and vice versa. Setting up a mailserver is certainly not an easy task, it's quite easy to turn it into an open relay that's going to be abused by spammers. But if you keep it only on your local network it can't do any harm and you will be able to try various different setups.
That's how most of my home network started. Get a few surplus machines, doesn't need to be powerful machines, any old thing with a bit of memory and some harddisk space will do fine. Read a lot of documentation and just started hacking away. I would start by learning about the SMTP protocol. That's the most important protocol for email. It's a fairly simple protocol, it's not difficult to understand.