What's the significance of Robots.txt file in a website?
webguru_1
-8
Newbie Poster
Banned
Recommended Answers
Jump to PostTo tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.
Jump to PostThe irony is palpable.
All 5 Replies
Dani
3,986
The Queen of DaniWeb
Administrator
Featured Poster
Premium Member
pty
882
Posting Pro
AndrisP
commented:
Webguru asks this question?
+6
Vikash_12
0
Newbie Poster
agustina
0
Light Poster
sandy_5
-7
Junior Poster in Training
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.