Member Avatar for Lampard33

Valid meta robots content values
Googlebot interprets the following robots meta tag values:
* NOINDEX - prevents the page from being included in the index.
* NOFOLLOW - prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
* NOARCHIVE - prevents a cached copy of this page from being available in the search results.
* NOSNIPPET - prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
* NOODP - blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
* NONE - equivalent to "NOINDEX, NOFOLLOW".
A word about content value "NONE"
As defined by robotstxt.org, the following direction means NOINDEX, NOFOLLOW.
<META NAME="ROBOTS" CONTENT="NONE">

  • 3 Contributors
  • 2 Replies
  • 111 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by stymiee

Dani AI

Generated

's post is a solid baseline for older robots meta behavior and is right to point to Google as the authoritative source. Search engines and directives have evolved since 2007, though, so the practical rules below update and expand that baseline. For the current, official reference see Robots meta tag and X‑Robots‑Tag.

Practical points to apply right away:

  • To remove or control non‑HTML files (PDFs, images, feeds) use the HTTP header approach rather than an HTML meta tag. Example header:
    X-Robots-Tag: noindex, nofollow
  • Do not rely on robots.txt blocking when your goal is to remove a URL from Google’s index. If a page is disallowed by robots.txt Googlebot can’t fetch it and therefore can’t see a meta noindex — the URL can remain indexed without content. Allow crawling and use a meta noindex or X‑Robots‑Tag, or serve a 404/410 for definitive removal.

Link handling and modern changes:

  • Page‑level “nofollow” differs from per‑link rel="nofollow"; page‑level prevents following links on that page, while the link attribute targets a single link. Since 2019 Google treats rel="nofollow" as a hint for crawling/ranking and introduced rel="ugc" and rel="sponsored". See Google’s blog on the change for details: .

Troubleshooting checklist (when results don’t match expectations):

  • Use Search Console’s URL Inspection to see what Google sees.
  • Check HTTP headers for X‑Robots‑Tag and ensure the page returns the expected status code.
  • Remove conflicting signals (e.g., disallow in robots.txt and a noindex meta on the same URL).
  • For urgent removals, consider the temporary removal tool in Search Console, then fix the source directive for permanent removal.

If not using any directives (as noted), keep that approach unless you need fine control over indexing, snippet size, or link following; then use the targeted meta/header directives above.

Recommended Answers

All 2 Replies

Member Avatar for Dexter515

Don't think I use any of them..

Member Avatar for stymiee

Thank you for copying what Google has in the blog and posting it here.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.