Those who are still confused about robots.txt.

Here is a link from mattcutts.
http://www.mattcutts.com/blog/robots-txt-remove-url/

Recommended Answers

All 2 Replies

An XML Sitemap (upper-case) file, which you can submit through Google's Webmaster Tools, makes it easier for Google to discover the pages on your site.

Using a Sitemap file is also one way (though not guaranteed) to tell Google which version of a URL you'd prefer as the canonical one.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, learning, and sharing knowledge.