Google frowns upon multiple pages with the same content. For example, if two different URLs have the exact same content on them, google considers it spamming their search engine. This forum uses Apache's mod_rewrite to rewrite URLs to have a .html extension for search engine purposes. Therefore, the webpage showthread.php?t=100 is the exact same thing as thread100.html - if google spiders see this duplicate contact, they will think that daniweb.com is trying to inflate its page count in google by having multiple URLs with the same content. However, by using robots.txt to block google from spidering the showthread.php pages, google only spiders the pages ending in .html - and therefore doesn't penalize us for duplicate content.