Member Avatar for jaimehopkin 
    <meta name="robots" content="index,follow"/>
    <meta name="robots" content="noarchive"/>

    Please suggest me friends, i know the above script, but i want to know the below script...
  • 4 Contributors
  • 4 Replies
  • 128 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by Jobsikd

Dani AI

Generated

Good summary from on what those meta tokens mean. A few practical points and a short robots.txt primer to clear up common confusions for , and .

Meta robots vs robots.txt vs X-Robots-Tag: meta robots live in the page HTML and require the crawler to fetch the page to see them. robots.txt lives at the site root and tells crawlers what NOT to fetch. That means blocking a page with robots.txt can prevent a crawler from seeing a meta noindex on that page, so robots.txt is not a reliable way to force removal from search results. Use a meta noindex (or an X-Robots-Tag HTTP header for non-HTML files like PDFs) on a crawlable page to remove it from index. Also note index,follow is the default behavior for most engines — the explicit tag is usually unnecessary.

Robots.txt basics (create a file at /robots.txt; it must be in site root): the main directives are User-agent, Disallow, Allow (supported by major engines), and Sitemap. Crawl-delay and Host are non-standard (not supported by Google). Robots.txt is public, so do not list sensitive URLs there if you expect secrecy — use authentication or remove the file instead.

Example (minimal) robots.txt:

User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml

Troubleshooting and best practices: if URLs remain in search after adding noindex, check that robots.txt is not blocking crawlers from fetching the page, confirm the page returns a 200 and the noindex/meta header is present, and use your search engine’s URL inspection or removal tools to request re-crawl or temporary removal. Avoid relying on robots.txt to hide confidential data. Use the X-Robots-Tag for non-HTML assets and canonical tags to handle duplicates rather than blocking crawls.

Recommended Answers

All 4 Replies

Member Avatar for pritaeas

index/noindex - Store this page for indexing, or not
follow/nofollow - Follow the links in the current page, or not
archive/noarchive - Keep a cache of this page, or not

Member Avatar for jaimehopkin

Thanks to solve my problem pritaeas..now i got a clear idea on that...

Member Avatar for rickydiar

hi,
can any one tell me how to write Robots.txt file n major attribute of that.

Edited by happygeek because: spam deleted
Member Avatar for Jobsikd

I don't know anything about this... Is there anyone to tell me about that...

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.