Member Avatar for Dani

I thought you were supposed to put a link to your sitemap in your robots.txt (isn't that what you said, stymiee?)

My robots.txt now looks like the following:

Sitemap: http://www.daniweb.com/sitemap.xml
User-agent: *
Disallow:

Google Webmaster Tools robots.txt says "Syntax not understood" for the sitemap line.

  • 3 Contributors
  • 3 Replies
  • 81 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by stymiee

Dani AI

Generated

Short answer: pointing a robots.txt at a sitemap is an accepted, standard practice and is supported by the Sitemaps protocol and major search engines. was correct that engines look for a sitemap declaration, and ’s observation about a validator complaining is a sign to verify the file rather than to remove the declaration.

Practical checklist (most common fixes and best practices): make sure the sitemap location is a full, reachable URL; the robots.txt file is served from the site root, returns HTTP 200, and is plain text with no byte-order mark or stray HTML; each sitemap declaration appears on its own line; multiple sitemap declarations are allowed if you use more than one sitemap file. If a webmaster tool reports a syntax error, fetch the file directly in a browser, check response headers and encoding, and validate the sitemap URL itself. Submitting the sitemap directly to the search engine’s webmaster tool gives redundancy while you troubleshoot.

Authoritative references: the Sitemaps protocol describes using robots.txt for sitemap locations and rules for sitemap URLs (sitemaps.org protocol), and Google documents robots.txt and sitemap handling in Search Central (Google Search Central: robots.txt and Sitemaps overview).

Recommended Answers

All 3 Replies

Member Avatar for stymiee

You can. Google announced it in their blog. Looks like they haven't updated their webmaster tools to reflect that. I wonder where can inform them of it?

Member Avatar for Candy_ME

Oh!!! This information is really helpful. Does Yahoo have a nofollow facility?

Member Avatar for stymiee

They say they do but many people also claim they haven't implemented it yet.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.