I found this in a few different places around the 'net - hopefully it's helpful:

Working with Album Art in Windows Media Player

Set your own Album Art for your media folders and Media Player for Windows XP (MPXP). If you don't like the album art or have a folder of downloaded music that you want to add art to:

• Simply select any graphic from your browser.
• Save it inside the folder you want, and call it Folder.jpg.

Now when you play any music that resides in that folder, the Media Player will automatically pick it up and display that as album art.