Member Avatar for mheidi

Does anyone know if it's possible to exclude the search engine bots by using a robots.txt file on a Coldfusion website just like an HTML site?
Thanks,
Heidi

  • 4 Contributors
  • 5 Replies
  • 209 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by gklandes
Member Avatar for GrimJack

From the definition:

The robots.txt file can be used to stop a search engine spider from crawling all or part of your site. Create a robots.txt file in the main folder of your site which will direct the search engine spiders what they may and may not search. Spiders generally look for this file before doing anything.

this seems to imply that it will. I know that it works for databases and such so I can see no reason that Coldfusion websites would be any different.

Member Avatar for mheidi

Thanks so much for your response. I just wasn't sure if the robots.txt worked for databases.

Member Avatar for gklandes

Does anyone know if it's possible to exclude ... on a Coldfusion website just like an HTML site?...

It might help to clarify that Coldfusion generated pages ARE html regardless of the ".cfm" extension. Coldfusion is simply a tool for dynamically generating the HTML (or XML or even PDFs and images, for that matter). CF on the server creates the content and then sends it out to the browser which interprets it in the same way that it interprets any other data sent from a server. In that respect CF is like PHP, ASP, ROR, etc. They all create HTML on the server.

The robots.txt file is one way to do it. You can also include a META tag in the HEAD of your page.

<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
<body>...</body>
</html>
Member Avatar for nherbal

Just visit google webmaster tools & generate robots.txt for the same

Member Avatar for gklandes

Google WM Tools is a good but a quick fix. Eventually you will want to understand how they work and how to use them more effectively. You can get a good grounding in them at ...
http://www.robotstxt.org/

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.