Member Avatar for vectro

I can't find a section related to .htaccess, so I'm posting this here.

Bing is going crazy trying to index proxied pages from PHPRoxy. With Glype, it's easy to exclude browse.php in robots.txt. However, PHPRoxy doesn't use browse.php. It uses index.php. For example: /index.php?q=aHR0cDovL21pbmRjdHJs

I know it's possible to include user agents in .htaccess code. How can I use .htaccess to prevent Bing from crawling index.php?= (but without blocking index.php entirely) no matter what string comes after the = symbol?

  • 2 Contributors
  • 3 Replies
  • 89 Views
  • 2 Weeks Discussion Span
  • Latest Post Latest Post by pritaeas

Dani AI

Generated

raised a good point: you only want to stop Bing from indexing requests like index.php?q=... without disabling index.php itself. was right that robots.txt can help, but robots rules are path-based and cannot reliably target only requests that include a query string. A server-side rule is the reliable way to target requests with query strings from a specific user agent.

Two practical .htaccess approaches:

  • Return a 403 for Bing when index.php is requested with a non-empty query string (hard block).
  • Send an X-Robots-Tag: noindex, nofollow header for Bing on those requests (allow fetching but instruct the crawler not to index).

Example .htaccess snippets (adjust path and enable the modules mod_rewrite and mod_headers):

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} bingbot [NC]
RewriteCond %{REQUEST_URI} ^/index.php$ [NC]
RewriteCond %{QUERY_STRING} .
RewriteRule ^ - [F,L]

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} bingbot [NC]
RewriteCond %{REQUEST_URI} ^/index.php$ [NC]
RewriteCond %{QUERY_STRING} .
RewriteRule ^ - [E=NOINDEX:1]

Header set X-Robots-Tag "noindex, nofollow" env=NOINDEX

Notes and troubleshooting: test with curl using a Bing user agent to confirm the response or header, check your access logs to capture the exact Bing UA strings (add older variants if present), and remember that search engines may take time to drop already-indexed URLs. If you use CDNs or caching layers, ensure these rules run at the origin or that headers are preserved.

Recommended Answers

All 3 Replies

Member Avatar for pritaeas 
user-agent: bingbot
disallow: index.php
commented: Thanks for the help +2
Member Avatar for vectro 
user-agent: bingbot
disallow: index.php

I see what you're getting at. robots.txt can probably accomplish the task. If this forum had a 'Thanks' button I'd use it.

Edited by vectro because: n/a
Member Avatar for pritaeas

You can upvote posts, should you want to, using the arrows next to the post.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.