View Single Post
Jun 14th, 2004
0

Re: Using a robots.txt to block links

The robots.txt file I'm currently using is in my root directory (not my forum root) and looks like this:

User-agent: googlebot
Disallow: /techtalkforums/announcement.php
Disallow: /techtalkforums/faq.php
Disallow: /techtalkforums/forumdisplay.php
Disallow: /techtalkforums/login.php
Disallow: /techtalkforums/member.php
Disallow: /techtalkforums/newreply.php
Disallow: /techtalkforums/newthread.php
Disallow: /techtalkforums/online.php
Disallow: /techtalkforums/printthread.php
Disallow: /techtalkforums/search.php
Disallow: /techtalkforums/showthread.php

I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?
Administrator
Staff Writer
Reputation Points: 1422
Solved Threads: 162
The Queen of DaniWeb
cscgal is offline Offline
13,645 posts
since Feb 2002