What's the significance of Robots.txt file in a website?

Recommended Answers

To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.

Jump to Post

The irony is palpable.

Jump to Post

All 5 Replies

To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.

The irony is palpable.

commented: Webguru asks this question? +6

The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.

Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of 1.20 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.