Member Avatar for jwhite68

I have used mod_rewrite in .htaccess (on Apache web server) so that dynamic pages are treated as static html.

The mod_rewrite itself is working fine, so where I had 4 parameters on the URL (php scripts), its now a single html file, eg. product-1-2.html.

However, whilst most of my website is indexed by Google, none of the mod_rewrite html pages are indexed - therefore defeating the objective of using it in the first place.
Does anyone face similar issues, or have any tips / things to check?

I assumed that because it works, ie entering the html based URL does run the dynamic php page and display the correct content - that there was nothing more to it.

Any help appreciated.

  • 2 Contributors
  • 2 Replies
  • 109 Views
  • 21 Hours Discussion Span
  • Latest Post Latest Post by jwhite68

Dani AI

Generated

A few likely causes (and a focused way to diagnose) beyond the basic rewrite itself: search engines index the URL they consider canonical, and that choice can be changed by HTTP headers, meta tags, redirects, server responses or by which URL is linked from your site. already flagged crawl access; has confirmed the site serves the .html URLs to visitors. The next steps are to see how crawlers actually see those pages and to force a single canonical URL if necessary.

Quick diagnostic checklist:

  • Use Google Search Console URL inspection to fetch the .html URL and check the chosen canonical and index status.
    Google Search Console

  • Check HTTP response headers for the .html URL to confirm a 200 OK and absence of X-Robots-Tag or unexpected redirects:

    curl -I "http://yourdomain.com/property-1-2.html"

  • View page source for any meta robots noindex or a rel="canonical" that points away from the .html URL. If a canonical tag points to the dynamic script, Google will prefer that.

    <link rel="canonical" href="https://yourdomain.com/property-1-2.html" />

  • Inspect server logs for Googlebot requests and response codes, and run a site crawl (Screaming Frog, Xenu) to confirm internal links truly point to the .html URLs (not JS-only links).

If Google favors the dynamic URL, fix it by setting the .html as the canonical (or issuing a server-side 301 from the dynamic URL to the .html), submit a sitemap that lists the .html pages, and request reindexing via Search Console. The .htaccess deny for the .htaccess file and Options -Indexes are normal server hardening and should not prevent indexing of rewritten pages.

Recommended Answers

All 2 Replies

Member Avatar for Dani

Make sure that you're not using a robots.txt file that might be disallowing spiders to have access to the files. Also make sure you are linking to the mod_rewritten versions throughout your website. If the dynamic URLs are already indexed in the search engines, it might take some time to tell the SEs you want them to use a different URL to access the same page instead. Try using a 301 Redirect from the dynamic versions to their static counterparts.

Member Avatar for jwhite68

Heres a part of my .htaccess showing the rewrite rule:

RewriteRule ^property-([0-9]*)-(.*).html /fullpage.php?id=$1&mode=prop&redir=$2 [L]

So, it should be able to take property-1.html and convert that to the php (dynamic) equivalent.

Within my website code, I refer to the links as .html too, as per your suggestion. I can also see that the SEs are not indexing the dynamic versions, nor is there anything in robots.txt to prevent them being indexed.

I do have this within my .htaccess though, and not sure exactly what it does or whether its correct:

DirectoryIndex index.php
Options -Indexes
<Files .htaccess>
order allow,deny
deny from all
</Files>

Any other thoughts?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.