Under what circumstances would you look to exclude pages from search engines using robots.txt vs meta robots tag?

8 Years
Discussion Span
Last Post by diafol

Pages you didn't want the search engine to index.

Okay, to be more specific, log-in pages to admin sections of a site.
Any other admin type pages, say for a CMS.
Members areas that might include a set of tutorials or pdfs or private info of some sort.


Or are you asking which method to use?

AFAIK - neither will stop robots determined to sweep your entire site - only the well-behaved ones. But there again, you're talking SEO not protection right? Or the other way around?

Your cgi-bin obviously, login pages, form handlers, includes folders, images etc etc. I *usually* set a robots.txt file to disallow quite a few complete folders. HOWEVER, this is publically viewable, so the paths of your hidden files/folders etc can be known. Using the meta tag (nofollow, noindex) is another 'useful' addition.

Ensure that all 'hidden' files are protected so that they cannot be accessed without a secure login etc, e.g. bump the robot/user to a default page if they try to access without authorisation.

Edited by diafol: n/a

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.