Does anyone know if it's possible to exclude the search engine bots by using a robots.txt file on a Coldfusion website just like an HTML site?
Thanks,
Heidi
mheidi
0
Newbie Poster
Recommended Answers
Jump to PostFrom the definition:
The robots.txt file can be used to stop a search engine spider from crawling all or part of your site. Create a robots.txt file in the main folder of your site which will direct the search engine spiders what they may and may not search. Spiders generally …
All 5 Replies
GrimJack
1,414
Posting Maven
Featured Poster
From the definition:
The robots.txt file can be used to stop a search engine spider from crawling all or part of your site. Create a robots.txt file in the main folder of your site which will direct the search engine spiders what they may and may not search. Spiders generally look for this file before doing anything.
this seems to imply that it will. I know that it works for databases and such so I can see no reason that Coldfusion websites would be any different.
mheidi
0
Newbie Poster
Thanks so much for your response. I just wasn't sure if the robots.txt worked for databases.
gklandes
Does anyone know if it's possible to exclude ... on a Coldfusion website just like an HTML site?...
It might help to clarify that Coldfusion generated pages ARE html regardless of the ".cfm" extension. Coldfusion is simply a tool for dynamically generating the HTML (or XML or even PDFs and images, for that matter). CF on the server creates the content and then sends it out to the browser which interprets it in the same way that it interprets any other data sent from a server. In that respect CF is like PHP, ASP, ROR, etc. They all create HTML on the server.
The robots.txt file is one way to do it. You can also include a META tag in the HEAD of your page.
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
<body>...</body>
</html>
nherbal
0
Newbie Poster
Just visit google webmaster tools & generate robots.txt for the same
gklandes
Google WM Tools is a good but a quick fix. Eventually you will want to understand how they work and how to use them more effectively. You can get a good grounding in them at ...
http://www.robotstxt.org/
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.