What is Robot.txt in SEO?
chrispitt
-2
Light Poster
Recommended Answers
Jump to PostIt is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.A robot.txt is a text file …
Jump to PostThe file goes in the root directory of your website, yes.
All 7 Replies
Octet
45
Newbie Poster
Featured Poster
It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.
A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.
In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.
ravinder yadav
commented:
thanks for answer but do reply my quary
+0
allenhill99
0
Newbie Poster
robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.
semseogurus
0
Newbie Poster
It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.
In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.
Yes, i am agree with your answer.
almostbob
866
Retired: passive income ROCKS
note the corrections in allenhill99 & creditcards replies
robot.txt is nothing
the file must be named robots.txt
ravinder yadav
0
Newbie Poster
but where is this file attached? is it attached with index file(i mean is it in root directory of our site?)
Octet
45
Newbie Poster
Featured Poster
The file goes in the root directory of your website, yes.
ravinder yadav
commented:
thanks
+0
bhav123
0
Junior Poster in Training
A robots.txt file restricts access to your site by search engine robots that crawl the web. It is a text file that tells search engine spiders also known as search engine robots, which parts of your website they can enter and which parts they can’t.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.