Member Avatar for Dani

We have the valid URLs:

www.daniweb.com/foo
www.daniweb.com/foo/
www.daniweb.com/foo/1
www.daniweb.com/foo/2
www.daniweb.com/foo/3

If I want to disallow them all in robots.txt, are both of these valid and will they do the same thing?

Disallow: /foo
Disallow: /foo/

Will the latter also block the URL www.daniweb.com/foo or will that be interpreted as a page underneath the root directory, and not within the foo directory? Contrastly, will the former be interpreted as only blocking the single page and not the foo directory?

  • 2 Contributors
  • 1 Reply
  • 248 Views
  • 2 Hours Discussion Span
  • Latest Post Latest Post by CimmerianX

Dani AI

Generated

Short answer: under the current Robots Exclusion Protocol the path in a Disallow rule is treated as a prefix, so a rule that begins with the same characters as a directory will match that directory and its children. This is defined in the standard (RFC 9309, published September 2022). (RFC 9309)

's question and 's intuition are normal — historically different robots behaved inconsistently — which is why many engines (including Google) explicitly recommend using a trailing slash when you mean “the whole directory” (it improves clarity and avoids accidentally matching similarly named paths). Google also documents wildcard (*) and end-of-URL ($) support and explains that when rules conflict the most specific (longest) path is chosen. (Google: useful rules and interpretation)

Practical notes and gotchas: percent-encoding and path normalization matter for matching (RFC 9309 describes the normalization rules). Some crawlers may be more lenient or implement extra features; if portability matters, be explicit (use the directory-slash form to show intent). Also remember that Disallow stops crawling but does not reliably prevent indexing — a blocked URL can still appear in search results if other sites link to it. To prevent indexing use a noindex meta tag or an X-Robots-Tag: noindex HTTP header (these require the page to be crawlable so search engines can see the directive). (RFC 9309, Google: block indexing and X-Robots-Tag, Google: robots meta / X-Robots-Tag)

Example approach (clarify intent, then test with Search Console or local tools):

User-agent: *
Disallow: /private/

Use tools (Search Console robots report or a local parser) to validate before and after deployment. (Google: create and submit robots.txt)

Member Avatar for CimmerianX

Using "Disallow: /foo/" would block the foo directory and everything in it.

Technically, without the trailing slash, Disallow blocks the 1 item, such as a single file. I would assume this would indicate a disallow on a single file named foo not the directory /foo/.

You can also use "Disallow: /foo*/" to block any subdirectory that began with "foo".

BTW, google has some webmaster tools available that will test the robots.txt file and report on the results. http://www.google.com/webmasters/

commented: Thanks! +15
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.