Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?

Recommended Answers

All 5 Replies

I personally never had such a file and I have not seen an impact on my site. Have others?

The robot.txt file is kind of obsolete, or at least fairly redundant, if you use XML Sitemaps.

Hi jiten,
When a search engine crawler or spiders visit your site then at first they search Robots.txt file.This file tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. If there is a Robots.txt file, then by default it will try to crawl all the pages,then it will ensure that none of the pages are skipped.

With Help of the Robots.txt you can guide the Search Engine which page you want to crawl and not crawl.

Geez Dave,
its more than a year old let the thread DIE

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, learning, and sharing knowledge.