Recommended Answers
Jump to PostRobots and spiders crawl your site. The file is there to give these spiders instructions once they visit your site. http://www.seoconsultants.com/robots-text-file/
Jump to Postrobots.txt is a text file which can be used to restrict web robots to accessing your web site only in ways of which you approve.
The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
Jump to PostThe robots.txt is a standard which was developed in 1994, when large-scale web indexing became popular; indexers such as Lycos and AltaVista used it
Jump to Postrobots.txt is mainly use for leave those pages which you not want to read by Google...
All 41 Replies
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.