23
Contributors
23
Replies
76
Views
4 Years
Discussion Span
Last Post by rohit03catalyst
-1

By the help of robots.txt files that Google and other search engines are crawling and indexing your website site.

-1

hey robot.txt is a normal text file. u can get snippet from google, it is very small and easy to use. robot. After creating tis file store it in www root folder. there are some portions on your website that you don't want google scrawler to scrawl so u can use this in meta tag. so, with the help of this robot file you can control the scrawling by Google.

0

Robot.txt is just a text file. It is used to tell search engines not to index some pages of a site.

-1

Robot.txt file is specially made for the the purpose of hiding some confidential data from the crawler so we can save or adim like things.

-1

Hi friend, Now that you know what is robots.txt file I am going to tell you what the robots are.

Robots are small programs to do some automatic work (Like the real robots or smart electronic toys). In search engine optimization robots are used to gather data/information from various data stores (websites etc.)
Google and other search engines use the robots (also known as spiders or bots) to gather all the information on the website (all the web pages and other stuff like pdf files).
When the bots come to any domain(website) they look for the instructions for them. We can feed them the instructions with the help of the robots.txt file. We can tell them what resources (pages/files) to cache (to copy the pages and files from your site and store them in their own servers to show them to the searchers searching for the similar information) and what we want to hide from direct search engine traffic.
Tell me if you are asking for this or something else. I will be happy to help you. Thanks.

-1

The Robots.txt is an easy textual content file within your website which notify search engine crawlers how to crawl and index website or webpages.

-1

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Google specifically follows instructions given in this robots.txt file

-1

A robots.txt file restricts access to your website page or folder by search engine robots that crawl the web. The bots are automated, and before accessing pages of a site they check if a robots.txt file exists that prevents them from accessing certain pages.

0

Robot.txt is a file which control search engine To visit website.We can made it easily. We should upload the robot.txt file in domain root folder.

-1

Robots.txt is helps to hide the pages from search engine to crawle. for example if we need to hide admin pages from google, we shold include the page in robts.txt file and upload into root folder.

0

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

0

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages. If you search in Google you will get sample for robot.txt

0

It's txt file that has details for crawlers and robots to say that what to index or what not.

0

robot.txt is a file that gives deatils to search engine which files and webpages you crwaled or not for particular search engine.

0

The robots.txt is a simple text computer file in your web site that inform online search engine crawlers how to spider and index website or web pages.
By default online search engine crawlers spider everything possible unless they are forbidden from doing so. They always scan the robots.txt computer file before crawling the web site.

0

robot.txt is a normal text file. u can get snippet from google, it is very small and easy to use. robot. After creating tis file store it in www root folder. there are some portions on your website that you don't want google scrawler to scrawl so u can use this in meta tag. so, with the help of this robot file you can control the scrawling by Google.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.