Member Avatar for vectro

I need mod_security rules to prevent Googlebot from indexing any file named browse.php anywhere on the server, while still allowing Googlebot to access anything else. I figured mod_security will do the trick because it can recognize user-agents and set rules accordingly.

Any ideas?

  • 1 Contributor
  • 1 Reply
  • 205 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by vectro

Dani AI

Generated

Good call, — your LocationMatch+SecRule approach is a simple, effective way to stop a crawler at a specific path. A couple of practical alternatives and cautions that make the behavior more robust and easier to test.

If the goal is "prevent indexing" (not necessarily block access), prefer sending a per-request X-Robots-Tag header to Googlebot only. That lets Google crawl the file but drop it from results (safer if other automation depends on the URL). Example (Apache, requires mod_setenvif + mod_headers):

<FilesMatch "(?i)browse\.php$">
  SetEnvIfNoCase User-Agent "Googlebot" is_googlebot
  Header set X-Robots-Tag "noindex" env=is_googlebot
</FilesMatch>

If you really want to refuse Googlebot fetches (deny crawling), a server-wide ModSecurity rule that matches the request URI anywhere and then checks the User‑Agent is a cleaner pattern than only using a root LocationMatch. Example ModSecurity (v2) pattern:

SecRule REQUEST_URI "@rx (?i)/browse\.php($|\?)" "phase:1,chain,id:950001"
SecRule REQUEST_HEADERS:User-Agent "@pm Googlebot" "deny,status:403,msg:'Block Googlebot on browse.php'"

Testing and troubleshooting: curl with a Googlebot UA to verify behavior, and also test with a normal UA to ensure no collateral blocking. Watch your ModSecurity audit/log file and Apache error log while testing. Remember that UA checks are easy to spoof — if you must be certain the client is genuine Googlebot, implement reverse‑DNS + forward lookup verification (costly per request).

Final notes: robots.txt disallow will prevent crawling but not necessarily remove the URL from search results. Pick X-Robots-Tag (noindex) if you want removal without outright blocking; pick ModSecurity deny if you want to stop fetches entirely.

Member Avatar for vectro

I did some research on creating mod_security rules and figured this out myself. Here is a server-wide mod_security rule for the main Apache configuration which will keep Googlebot off of 1 particular file. This only applies if the file shows up in the root directory of a domain, but it applies to all domains on the server.

<LocationMatch "/file.php">
SecRule REQUEST_HEADERS:User-Agent "@pm Googlebot" "deny,status:403"
</LocationMatch>

Change file.php to the name of the file you want to protect. The part that says "Googlebot" can also be changed to any user-agent. It's a pattern match and not an explicit match. This means the full user-agent simply needs to include the word for the rule to apply.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.