I sent you a PM RE: this topic.

And I started this thread RE: your PM

Ok I am a knuckelhead. I guess this was a chicken before the egg thing. ;-)

The robots.txt file I'm currently using is in my root directory (not my forum root) and looks like this:


I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?

without going too far in, it looks decent to me. I am not sure however, if the robots.txt blocks the weakening link pop from all the links. The less links on a page the more potent the links are. A page with tons of links is spreading the pop thin. That would be a good question to ask SEO-Guy.

It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)

*nods* for sure.

I don't see how this will help at all.

Google frowns upon multiple pages with the same content. For example, if two different URLs have the exact same content on them, google considers it spamming their search engine. This forum uses Apache's mod_rewrite to rewrite URLs to have a .html extension for search engine purposes. Therefore, the webpage showthread.php?t=100 is the exact same thing as thread100.html - if google spiders see this duplicate contact, they will think that daniweb.com is trying to inflate its page count in google by having multiple URLs with the same content. However, by using robots.txt to block google from spidering the showthread.php pages, google only spiders the pages ending in .html - and therefore doesn't penalize us for duplicate content.