Which one is better Robots.txt or Meta robots Tag

[samuelmax]

Which one is better Robots.txt or Meta robots Tag ?

[Dani]

It depends what you're trying to do. We use <meta name="robots" content="noindex" /> when we want Google to crawl a page (and follow outgoing links on it), but not index the page itself (because it's thin content, for example). We use robots.txt to block pages from Google that are unavailable to non-logged in members, and therefore have no content at all to show, or that we don't want Google to even look at for a particular reason.

Keep in mind that the amount of Pagerank that you have is finite, depending on the quality and quantity of your backlinks. Pagerank determines how often and how deep Googlebot is willing to crawl your site. Therefore, you may not wish to waste your Pagerank having Googlebot spidering and indexing mediocre pages of your site, and never getting to your important pages.

Therefore, the whitehat/greyhat technique of Pagerank sculpting is used to block Googlebot from the pages you don't care much about, to ensure that only the pages you do care about get indexed.

We currently use the meta robots tag for member profiles for newbies. We do this because we have just about one million members, and there's really no need to waste our Pagerank having Google spider and index so many profiles from members who have no posts.

We also use the meta robots tag for Q&A discussion threads that have no replies yet, because those are unhelpful to people coming in from a Google search. Once they receive a reply, we remove the meta robots tag. We are able to do this because we have enough Pagerank that Googlebot comes and revisits the same page in a timely manner.

We block pages such as the "start new discussion" page, edit profile pages, and even the entire Business Exchange category (only available to logged in members), through robots.txt, because all that Googlebot (or any other non-logged in member) would ever see is an error message.

[samuelmax]

thanks dani. your this post means me alot. keep helping and suggest me like this i need help in future too from you.

[audrey110]

i think dani have gave clear view of theam....thanks dani

[getxhtml]

Both are similar as robots txt is used to stop web crawler not to crawl particular page or content as Meta robots effort is as well same. The main dissimilarity is of structure how to use them both. Working of both robots.txt and Meta robots are same.

[jyotsna01]

robot.txt is better..
easy to use and more effective than meta robots tag..

[vkmigrate]

I think meta robots is best and power full way to disallow any file or folder.