What's the significance of Robots.txt file in a website?

To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.

The irony is palpable.

commented: Webguru asks this question? +6

The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.

Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.