Member Avatar for fumnimda

hi,
i have a robots.txt file with this content:
User-agent:*
Allow: /
Allow: /sitemap
Disallow: /gerente

is it written in right order?

  • 3 Contributors
  • 2 Replies
  • 87 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

Quick clarification for : crawlers do not rely on the visual order of lines the way people do. What matters is which user-agent group a crawler matches and which path rule is the most specific. Overlapping Allow/Disallow rules can therefore be ambiguous; use separate user-agent blocks for different bots or make path patterns explicit so the most specific match is unambiguous. This is how major engines resolve conflicts — see Google's parser rules for details (Google's Robots.txt documentation). As suggested, reading the official parser guidance helps; and 's note about including sitemap info is fine but make sure the sitemap URL you point to is reachable.

Quick practical checklist and troubleshooting tips:

  • Confirm your site serves a single /robots.txt from the site root and that it returns HTTP 200 (not a redirect or 404).
  • Use Search Console's robots tester and Fetch/Render to simulate Googlebot and verify that pages and resources are accessible.
  • Avoid blocking CSS/JS or other resources required to render pages — that can harm indexing.
  • If the goal is to prevent a page from appearing in search results, use a noindex meta tag or X-Robots-Tag; robots.txt only blocks crawling and does not reliably prevent indexing of linked URLs.
  • If you use wildcards (*) or end-of-line anchors ($), test across the engines you care about.

For sitemap format and submission details, consult the Sitemaps protocol (sitemaps.org/protocol.html). After any edits, request a recrawl and monitor coverage reports and server logs to confirm crawler behavior.

Recommended Answers

All 2 Replies

Member Avatar for AirForceOne

Hi,

I think these 2 lines would be not useful Allow: / Allow: /sitemap for any purpose since they are allowed by default. Also, might I suggest the contents on robotstxt.org/orig.html#examples for you to read.

all the best,

Member Avatar for Dani

My robots.txt file looks like this:

Sitemap: http://www.daniweb.com/sitemap.xml

User-agent: Googlebot
Disallow: ...........

It sits in the root directory. I think that Allow is explicit.

Edited by Dani because: n/a
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.