Member Avatar for phoenix06007

Google and Yahoo allow a Sitemap: <sitemap_location> line in the robots.txt.

Do any of the other search engines use that feature?
Does it this feature help?

  • 5 Contributors
  • 4 Replies
  • 93 Views
  • 2 Weeks Discussion Span
  • Latest Post Latest Post by 4nicefriend

Dani AI

Generated

asked whether other engines honor a robots.txt sitemap pointer; and recommended adding one, and reported a severe ranking drop after changing sitemap/robots. A robots-based pointer is a useful, widely-read signal but it is only one part of a healthy indexing workflow — it helps crawlers find the XML sitemap, but does not guarantee indexing or prevent ranking issues.

When using a sitemap pointer as suggested by others, follow these practical checks: confirm the sitemap file is reachable with a 200 response and not blocked by server rules; validate the XML against the sitemap protocol and split into an index when over 50,000 URLs or 50MB (uncompressed); submit the sitemap directly in each search engine’s webmaster tools and watch Coverage/Indexing reports; inspect server logs and crawl stats to spot 4xx/5xx problems or reduced crawl activity; and verify there are no site-wide noindex tags, canonical mistakes, or manual actions that would remove pages from results. Note that metadata like lastmod or changefreq are treated as hints by major engines.

For authoritative details and protocol limits, see Google Search Central’s sitemaps guide: Google Search Central: Sitemaps. If a site’s listings suddenly vanish after changes, prioritize checking Search Console messages, manual actions, and server/robots configuration before assuming the sitemap pointer is the root cause.

Recommended Answers

All 4 Replies

Member Avatar for sneegdha

I am going thru a very bad experience with google sitemap and updated robots.txt
Since last year my site's listing has almost vanished from google and alexa rank of 60000 has come down to 300000

Member Avatar for hotelaudit

If you place your sitemap between
<sitemap_location> </sitemap_location> command is perfect for the all search engines.

Member Avatar for x3mario

I agree with hotelaudit. That would work for all SE's.

Member Avatar for 4nicefriend

If you place your sitemap between
<sitemap_location> </sitemap_location> command is perfect for the all search engines.

Ok, I'll try..

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.