Dear All,

I want google bot to visit all my web pages. What type of robot.txt code should be used to allow google bot visit all my webpages?

Recommended Answers

All 6 Replies

You don't need a robots.txt file if you don't want to disallow robots from anything. However, it can't hurt to create one to specify your sitemap file:

Sitemap: http://URL-TO-SITEMAP-FILE

First check your robots.txt file and make sure you have the command:
User-agent:*
Allow:/

That would override any disallow directives you may have defined, but only for those web crawlers that observe the Allow directive. Others could still be blocked.

That's why I say it's better to just be blank.

You need a robots.txt file only if your site includes content that you don't want Google or other search engines to index. To let Google index your entire site, don't make a robots.txt file (not even an empty one).

Not even an empty one? But what if the site contains content that is not discoverable through the normal link crawling process?

Providing a minimal robots exclusion file with just a sitemap reference, as Dani first suggested, will help to ensure all pages are crawled. Even the ones that aren't linked.

I realize the OP is only asking about Google. Yes, it's possible to notify Google about a sitemap though their Webmaster Tool, but generally speaking it's more practical to use the robots exclusion file. That way you only need to specify it once and every crawler will have the opportunity to discover it.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.