Welcome to a long frustrating journey! ;) Seriously, it is obvious you are just starting this journey so expect a lot of struggling. But soon, it will all come together--just keep pushing forward and learning.
First of all, PHP is not only a web development language. You can use PHP to write shell scripts as well as develop full GUI apps. Since you do not say, I have to assume you are learning PHP as a web development language.
As you've already discovered, you can not simply double click a PHP file on your local filesystem and have it process and display in your browser. You are probably familiar with doing this with HTML files--which does work.
PHP must be processed by the webserver. This means you need both a webserver as well as the php interpreter integrated somehow with your webserver (so the webserver knows how to process the PHP code before sending the results to the browser).
The good news is that it's relatively easy to set this up on your local computer. You have several options, but the two most popular are to install:
1. IIS and PHP
2. Apache and PHP
I recommend Apache and PHP for several reasons, but the 2 most important are:
1. Apache runs on both Windows and Linux, so the configuration skills you learn will work on either.
2. IIS is not available anymore for Windows XP Home and Pro. (I think....in any case, it's increasingly difficult to get a Windows webserver you can use on anything other than Windows Server version.)
Apache and PHP (along with MySql) on Windows is commonly referred to as "WAMP" (Windows, Apache, MySql, PHP). "LAMP" is the same thing but Linux instead of Windows.
So, my recommendation to you is to install Apache, then install PHP. You must then configure Apache to use the PHP module. There are many places to learn how to do this, but I'm partial to my own WAMP HOWTO at http://www.troywolf.com/articles/wamp_howto.htm
Now, my HOWTO is a bit dated, but still very relevant. It explains to install PHP5, but if I were to update it, I'd change it to PHP4.4.2 which has less compatiblity issues with some of the popular PHP application packages out there. For your purposes though, installing either PHP version will be just fine for your learning.
Once configured, to test your PHP pages, you'll surf to them in your browser using http://localhost/foo.php (or whatever you name your scripts).