Member Avatar for MattEvans

The index page of my site is redirected permanently to another page on my site using .htaccess, I'm messing about with IBP9's spider simulator to see how a search engine would see it; apperently this is what they'd see from the index page:

<html>
<head>
<meta http-equiv="refresh" content="0;url=./YaBB.pl">
</head>


I've changed the index page (it was totally blank) to have a hyperlink and note to the redirected page, still, the HTML above is all a spider sees.

If I spider the redirected page it's alot more fruitful, but most of my hyperlinks elsewhere (like the one in my signature :P) link directly to the root directory, so as they hit the redirect.

Is this ACTUALLY what the spider is seeing? and can I fix this problem without not using .htaccess?

Matt

  • 1 Contributor
  • 1 Reply
  • 128 Views
  • 12 Hours Discussion Span
  • Latest Post Latest Post by MattEvans

Dani AI

Generated

As discovered, the issue wasn’t .htaccess itself but the file the redirect pointed to. If your root URL is forwarded to an HTML page that uses a client-side redirect or a tiny placeholder page, a spider will often fetch and index that HTML instead of the final destination. That explains why IBP9’s simulator (and real crawlers) showed the minimal page content.

Prefer server-side HTTP redirects (301 for permanent moves). They present an HTTP status and Location header so crawlers follow and index the final URL, and they preserve SEO value better than meta-refresh or JavaScript redirects. To confirm what your server actually sends, inspect response headers. For example:

curl -I http://yourdomain.com/

Look for a 301/302 status and a Location header; use curl -L -I to follow redirects automatically.

Implement the redirect in .htaccess as a server-side redirect. A simple permanent redirect looks like:

Redirect 301 / https://example.com/targetpage

If you need pattern handling or to preserve query strings, use mod_rewrite with the R=301 flag and test thoroughly to avoid redirect loops.

Practical follow-ups: change internal links (including forum signatures) to point directly at the final URL so crawlers don’t hit the intermediate file; configure any spidering tools to follow redirects; watch server logs while testing; and request reindexing with Google Search Console’s URL Inspection. For guidance on how search engines treat redirects and HTTP status codes, see and MDN: HTTP redirections.

Member Avatar for MattEvans

Ahh.. the problem was simple, the .htaccess redirect went to an index.html page with the above html; I think that page was provided with YaBB; because I don't remember writing it.

Still, problem solved, I just changed the redirect.

Matt

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.