What's the significance of Robots.txt file in a website?

Recommended Answers

All 5 Replies

To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.

The irony is palpable.

commented: Webguru asks this question? +6

The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.

Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.