Member Avatar for mtyide

Hi. Just wanted to find out which option is best to use regarding SEO?

Edited by Dani because: Moved to SEO forum
  • 2 Contributors
  • 1 Reply
  • 54 Views
  • 19 Hours Discussion Span
  • Latest Post Latest Post by Dani
Member Avatar for Dani

Robots meta tags and robots.txt do two completely different things. Which you use depends upon the specific use case for each of your pages.

The robots.txt file is used to disallow certain bots (e.g. Googlebot) from accessing parts of your site. You can specify which files or folders should be inaccessible and which bot(s) they should be inaccessible to. (For example, you can make a URL accessible to Googlebot but not to Bingbot.) You may wish to make certain sections of your site uncrawlable by search engine spiders if you feel it would be a waste of resources to visit those sections. For example, Googlebot has limited crawl resources to devote to each website, known as crawl budget. Sometimes it makes sense to ensure Googlebot doesn't waste crawl budget crawling around an uninteresting section of your site, because then you might not have enough crawl budget left to crawl the high quality pages. Keep in mind that just because a webpage is inaccessible to Googlebot because of robots.txt, that doesn't mean that it might not appear in the search results anyways. Sometimes, if there are a lot of backlinks pointing to a URL, Google might decide that URL is valuable even if it can't access it! When that happens, there won't be a page title or description in the search results, and the URL will just appear on its own with no context.

You can use <meta name="robots" content="noindex"> when you want Google to not index a particular page. As long as the URL isn't blocked in your robots.txt, Google can access the page, and see and follow all the links on the page. It just won't include that page in the search results under any circumstances. Note that when there is a noindex meta tag on a website for a very long time, eventually Googlebot may stop visiting it entirely. When this happens, it won't see any changes you made to the page, which includes links on the page that you may have added/removed.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.