7
Contributors
7
Replies
53
Views
4 Years
Discussion Span
Last Post by bhav123
3

It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.

A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.

In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.

Edited by Octet

Votes + Comments
thanks for answer but do reply my quary
0

robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

-2

It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.

A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.

In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.

Yes, i am agree with your answer.

0

note the corrections in allenhill99 & creditcards replies

robot.txt is nothing
the file must be named robots.txt

0

but where is this file attached? is it attached with index file(i mean is it in root directory of our site?)

0

A robots.txt file restricts access to your site by search engine robots that crawl the web. It is a text file that tells search engine spiders also known as search engine robots, which parts of your website they can enter and which parts they can’t.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.