0

Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?

6
Contributors
5
Replies
6
Views
7 Years
Discussion Span
Last Post by devisrimari1
0

Hi jiten,
When a search engine crawler or spiders visit your site then at first they search Robots.txt file.This file tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. If there is a Robots.txt file, then by default it will try to crawl all the pages,then it will ensure that none of the pages are skipped.

0

With Help of the Robots.txt you can guide the Search Engine which page you want to crawl and not crawl.

Geez Dave,
its more than a year old let the thread DIE

0

hi,
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.