There's a large range of software you can use. PHPBB, Invision Powerboard, and VBulliten are the most popular currently. PHPBB and IPB are both free and offer god services. VBulliten is $160 (I think) and it's by far the best. You get the most control, best support, and best look. DaniWeb is a good example of a well done VB forum.
You'll need PHP (Usually 4.3 or higher) installed on your server and also a MySQL database. Sometimes the forums are slightly flexible with the type of database you use, I think some accept the Microsoft DB and other SQL variations.