Member Avatar for SubbuPD

Whats is the best statergy in writing a sitmap.xml and robots.txt?

  • 4 Contributors
  • 3 Replies
  • 69 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by vishnuvis

Dani AI

Generated

A short, practical checklist that fills gaps in the thread and helps avoid common pitfalls.

Sitemaps: list only canonical, reachable URLs and keep XML files UTF-8 with ISO 8601 dates. A single sitemap is limited (50,000 URLs / 50MB uncompressed), so use gzipped files and a sitemap index for large sites. Use sitemap extensions for images, video, or hreflang when relevant and update <lastmod> only when content truly changes. For full spec and examples see Sitemaps - Google Search Central.

Robots.txt vs indexing: robots.txt controls crawling, not whether a URL appears in search results. To prevent indexing, serve a meta noindex or an X-Robots-Tag header and allow bots to fetch the page (do not block it in robots.txt). Include a Sitemap: line in robots.txt so crawlers can find your sitemaps, and avoid disallowing CSS/JS that the crawler needs to render pages correctly. See official guidance at .

Practical workflow: define which pages should be indexed; ensure canonical tags and remove low-value parameterized URLs; generate sitemaps via a script for dynamic sites (or static files for very small sites); validate and submit sitemaps in Search Console; monitor coverage, server errors, and excluded URLs. Tools (as suggested) can help, but always review output; as warned, a bad config can remove pages from search. 's “keep it simple” robots approach is fine when it matches the index/crawl policy.

Recommended Answers

All 3 Replies

Member Avatar for AndreRet

You can use tools for creating the best site maps. Here are some tools.

Member Avatar for Dani

It completely depends on what type of website you have. Is it a static site with fewer than 50 pages? Or is it a dynamic site with millions of pages? If there are fewer than 50 pages, I strongly urge you to handwrite your sitemap file. If it's a dynamic site, then your best bet is to code a script to generate your sitemap (having database access is a must when determining which pages to index on a dynamic site.)

Once you have a strategy in place, then figure out what pages you want included in the sitemap, what pages to noindex, and what pages to exclude bots from crawling. This is something best left to experienced SEOs. If you're just getting started, we could help you with some ideas, or answer questions you may have, but I urge you to put a lot of thought into what goes here, because you could easily shoot yourself in the foot, and wind up deindexing all your pages from Google.

That's why I am very against using automated tools as AndreRet suggests. I think that hand crafting a sitemap.xml and robots.txt file are super important, and each line should never be taken lightly.

commented: You are totally correct Dani, if the OP gave us more information and some sign of effort, I would have elaborated as well. +14
Member Avatar for vishnuvis

Just keep the pages which you want to be indexed in google & remove the rest from sitemap. Robots.txt I follow the simple one unless there are some unnecessary query pages indexed.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.