Member Avatar for bellasteward

How important are robots.txt file?

  • 4 Contributors
  • 5 Replies
  • 107 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by MidiMagic

Recommended Answers

All 5 Replies

Member Avatar for surindersharma

Robots.txt is used to guide search engine through your website.

if you wish that a specific folder/webpages should not be crawled by an particular search engine. then robots.txt can be helpful for you.Robots.txt can control the excess of any search engine through out your complete website.

Member Avatar for bellasteward

Thanks surindersharma for replying to my post

Member Avatar for stephaniemcgrat

Robots.txt file controls crawler behavior. I am copy - pasting some lines from this article I found on Redalkemi site. It hope it is useful to you

Here are some tips on how to use robots.txt file -

1. The robots.txt file is always named in all lowercase (e.g. Robots.txt or robots.Txt is incorrect)
2. The robots.txt file is an exclusion file meant for search engine robot reference and not obligatory for a website to function. An empty or absent file simply means that all robots are welcome to index any part of the website.

Member Avatar for bellasteward

Hi Stephaniemcgrat..........thanks for the nice article. Its really very nicely explained and tips given in this article on redalkemi site, is really excellent

Member Avatar for MidiMagic

Note that some search engines can not search certain sites unless the robots.txt file is present.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.