What's the significance of Robots.txt file in a website?
webguru_1
-8
Newbie Poster
Banned
Recommended Answers
Jump to PostTo tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.
Jump to PostThe irony is palpable.
All 5 Replies
Dani
4,084
The Queen of DaniWeb
Administrator
Featured Poster
Premium Member
To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.
pty
882
Posting Pro
The irony is palpable.
AndrisP
commented:
Webguru asks this question?
+6
Vikash_12
0
Newbie Poster
The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.
agustina
0
Light Poster
Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.
sandy_5
-7
Junior Poster in Training
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.