Member Avatar for cmps

Hello all!!
I've been making websites for many years, and recently I decided to get more into SEO. So here's my question:
I would like to know what happens if I wrote in the robots.txt:

User-agent: *
Disallow: /
Sitemap:

And in the sitemap.xml, there are thousand of links linking to different pages in a directory, for example:

1 => http://www.example.com/site/index.php
2 => 
3 => 
4 => 
etc...

In this case, as I understand what robots.txt and sitemap.xml do;
First, robots.txt disallow Search Engine (Let's talk about Google) to index any file or folder in the domain name: example.com. However, google will look at the sitemap.xml and find that it has to index the concerned links.
What happens in this situation ?

Moreover I would like to know, what happens when a page has the meta robots set to noindex, but at the same time it appears on the sitemap.xml. What happens in this situation as well ?

Finally, I would like to know if I updated my path to the sitemap for my website on google webmaster tools, will it be enough for google to go and check it or it should also appear on the robots.txt ?

Thank you

Edited by cmps
  • 2 Contributors
  • 4 Replies
  • 257 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by cmps

Dani AI

Generated

Short answer: robots.txt controls crawling; a Sitemap is only a discovery hint. If you put Disallow: / but still list pages in your sitemap, Google won’t crawl those pages (it obeys robots.txt), but URLs can still appear in search as “URL-only” entries if Google finds links to them elsewhere. (developers.google.com)

If a page is blocked by robots.txt, Google cannot fetch it and therefore cannot see any on‑page meta robots or X‑Robots‑Tag headers. That means a noindex placed inside a page won’t be honored while the page is disallowed from crawling. To make noindex work, allow crawling long enough for Google to fetch the page, see the noindex, and then remove or change the rule. (developers.google.com)

You don’t have to put a sitemap entry in robots.txt if you submit it in Google Search Console, but adding Sitemap: to robots.txt is a harmless extra discovery method and is widely recommended. Do not include URLs in your sitemap that you intentionally block or mark noindex — that creates mixed signals and will produce warnings in Search Console. (developers.google.com)

Practical checklist (based on situation):

  • Decide whether you want the pages indexed or merely hidden from crawling.
  • If you want them out of search: allow crawling, add a page noindex (or X‑Robots‑Tag: noindex), wait for re‑crawl and verification in Search Console, then optionally re‑block. (developers.google.com)
  • If you want them hidden from crawlers but don’t care about search listings, keep them blocked in robots.txt but remove them from your sitemap to avoid confusing signals.
  • Use the URL Inspection and Sitemaps reports in Search Console to verify how Google sees each URL.

This preserves crawl budget, avoids contradictory signals, and gives predictable indexing behaviour.

Recommended Answers

All 4 Replies

Member Avatar for Member #949455

And in the sitemap.xml, there are thousand of links linking to different pages in a directory, for example:

I am bit confused with your sitemap.xml, I don't see an array like that before

It should look like this:

http://www.sitemaps.org/protocol.html

Member Avatar for cmps

@LastMitch you're totally correct, I just wrote this example to show links in the sitemap pointing to pages not allowed by the robot. And I was wondering what happens in such a conflict. Which one dominates ?

Here's a correction for the sitemap.xml code:

<url>
  <loc></loc>
  <lastmod>2013-10-12</lastmod>
  <changefreq>weekly</changefreq>
  <priority>1.00</priority>
</url>

 etc...

Thank you, I should have posted a better example of sitemap.xml in the first post

Member Avatar for Member #949455

And I was wondering what happens in such a conflict. Which one dominates ?

There's no conflict. robot.txt reads sitemap.xml if you put it in there.

The sitemap.xml has all the links on the website.

Member Avatar for cmps

Great, thank you for replying :)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.