What's the significance of Robots.txt file in a website?
webguru_1 -8 Newbie Poster Banned
Recommended Answers
Jump to PostTo tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.
Jump to PostThe irony is palpable.
All 5 Replies
Dani 4,675 The Queen of DaniWeb Administrator Featured Poster Premium Member
To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.
pty 882 Posting Pro
The irony is palpable.
AndrisP commented: Webguru asks this question? +6
Vikash_12 0 Newbie Poster
The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.
agustina 0 Light Poster
Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.
sandy_5 -7 Junior Poster in Training
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.