First write a program that compiles another program, probably by calling make.exe, nmake.exe, or some other such program that does command-line compiles on your computer. That will be the easiest part because all your program is doing is apawning another program.

The more dificult part will be running the test cases. Write the tests in an ordinary text file using notepad or some other editor that your program will read. Spawn the test program (the one your program just compiled) and stuff its keyboard buffer with the test information just like someone had typed it. That will be the somewhat challenging part, depending on what operating system the two programs run under. Then of course your program will have to learn how to detect what the test program did with the information.

My guestimate is that it will take you a year or so to write a useful program like that which can be marketed.

http://clusty.com/search?input-form=...+software+test

The more challenging aspects are:
- how to manage interaction
- how to deal with programs which exit with some kind of exception
- how to deal with programs which refuse to exit.

What is the form of the result?
- return status - easy
- output to a file - harder. Do things like "Program started at hh:mm:ss" count as a meaningful difference?
- output to a GUI on screen - tricky.