Member Avatar for zer0cool

Hi there,

Is there a way to stop robots from indexing a block of text on a page?

For example, I have a transcript on one of my pages that is not related to my website at all, and I don't want google or any other search engines to index or crawl that text.

Thanks in advance.

  • 3 Contributors
  • 2 Replies
  • 59 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by shaniadavid

Dani AI

Generated

— short answer: you cannot reliably apply a robots directive to only part of an HTML page. rel="nofollow" is for links, and meta/X-Robots-Tag directives apply to an entire resource, not a paragraph.

Practical, reliable options:

  • Move the transcript to its own URL (for example /transcripts/id.html) and mark that separate page noindex. Example meta tag to put in the head of the transcript page:

    <meta name="robots" content="noindex, noarchive">

    Or send the equivalent HTTP header:

    X-Robots-Tag: noindex, noarchive

    Important: do NOT block that transcript URL in robots.txt if you want search engines to see the noindex. If you disallow crawling, Google cannot fetch the page to see the noindex tag. If the content must never be indexed, consider requiring authentication or returning a 403/401 for non-authenticated requests.

  • If you need the transcript visible on your main page for users, embed it from the separate noindexed URL (iframe or server-side include). That keeps control over indexing while still showing content to visitors.

Notes and caveats: ’s client-side-loading idea worked against older crawlers, but modern search engines can render JavaScript, so it is not a dependable way to hide content. data-nosnippet or nosnippet will prevent text from appearing as a search snippet but will not stop indexing. As suggested, removing or hosting irrelevant material elsewhere often gives the cleanest, most controllable result.

Recommended Answers

All 2 Replies

Member Avatar for Tiggerito

Robots are not very good at processing javascript. So using javascript to dynamically load a block of text will probably block most if not all of them.

Member Avatar for shaniadavid

Well if it is not relevant to your website, Why you want that page to be with your website? I want to ask that what is the use of having a irrelevant page with your website?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.