Member Avatar for MattEvans

Is it true that spiders and bots don't like document-root or relative paths? I read it somewhere just now; and I don't know whether "don't like" means a spider will act as if there's nothing there and prompty leave; or feel somewhat offended, but compensate and carry on...

Do spiders go in circles? If all my pages have at least one "go back up one level" and lots of "go down some levels", are the bots clever enough to remember where they've been? If they're not... can I tell a spider to ignore just some links on a page?

  • 2 Contributors
  • 1 Reply
  • 124 Views
  • 10 Hours Discussion Span
  • Latest Post Latest Post by stymiee

Dani AI

Generated

Clarification for , building on 's point.

Modern search crawlers resolve relative and root-relative links against the page URL (or an explicit <base>). Relative paths are supported and will not by themselves cause a spider to “leave” or ignore a page. Absolute URLs are useful where the same content may appear under different hostnames or when content is syndicated, because they can reduce ambiguity, but they are not required for indexing.

Crawlers keep a queue and a set of seen/normalized URLs, so they do not endlessly refetch the exact same URL. The real issue is generating many distinct URLs that point to the same content (session IDs, tracking parameters, infinite prev/next chains). Those produce effective loops or waste crawl budget. Practical mitigations are canonicalization (rel="canonical"), 301 redirects to a single URL form, and avoiding link patterns that create unbounded URL sets.

Several ways exist to tell well-behaved bots to ignore links or pages:

<a href="/private/page" rel="nofollow">Private</a>
<meta name="robots" content="noindex, nofollow">
User-agent: *
Disallow: /private/
<link rel="canonical" href="https://www.example.com/path/page.html">
X-Robots-Tag: noindex, nofollow

Notes: robots.txt prevents crawling but does not always prevent indexing; meta noindex requires the page to be fetchable so the crawler can see the tag. rel="nofollow" is respected by major engines as a hint but cannot stop every bot. Recommended checks: inspect server logs and crawl stats, run a site crawler to map problematic link patterns, ensure base is correct, and pick one canonical URL format (http vs https, www vs non-www) then enforce it with redirects/canonical tags.

Member Avatar for stymiee

They don't mind relative paths at all. That's a myth and a poor one at that. It came about because Google recommends using full URLs in their webmaster tips and newbies instantly assumed relative URLs were bad. It's simply untrue.

You should use full URLs if you want to be 100% sure the search engines can follow every link (the chance for an error is .0000001% otherwise but some people like to be sure). It's also good in case someone scrapes your site. They usually screw it up and leave the full links up on their site. That then delivers traffic to your site and helps you find the scrapers.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.