RSS

Re: Using a robots.txt to block links

Please support our Search Engine Optimization advertiser: Affiliate Marketing

Using a robots.txt to block links

•

Join Date: Feb 2002

Location: Lawn Guylen, NY

Posts: 11,032

Reputation: cscgal is just really nice

Rep Power: 33

Solved Threads: 117

cscgal

Offline

The Queen of DaniWeb

Re: Using a robots.txt to block links

Jun 14th, 2004

The robots.txt file I'm currently using is in my root directory (not my forum root) and looks like this:

Help with Code Tags

(Toggle Plain Text)

User-agent: googlebot
Disallow: /techtalkforums/announcement.php
Disallow: /techtalkforums/faq.php
Disallow: /techtalkforums/forumdisplay.php
Disallow: /techtalkforums/login.php
Disallow: /techtalkforums/member.php
Disallow: /techtalkforums/newreply.php
Disallow: /techtalkforums/newthread.php
Disallow: /techtalkforums/online.php
Disallow: /techtalkforums/printthread.php
Disallow: /techtalkforums/search.php
Disallow: /techtalkforums/showthread.php

I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?

Dani the Computer Science Gal

DaniWeb Home > Forums > Internet Marketing > Search Engine... > Using a robots.txt to...

Re: Using a robots.txt to block links

Using a robots.txt to block links

Re: Using a robots.txt to block links