Setting up your own email server for the first time can be pretty tricky. Personally I like to use a Linux server and qmail with the addition of SPAMDYKE. There are some excellent scripts to setup the server for you and include anti-spam and anti-relay settings at the following web site:
The documentation is excellent and there are scripts for various versions of Linux.