Member Avatar for Karry.Stewart

Hi I've a website and it shows a strange urls on the search engine and it sometimes crawls or wont show in the search engines what to do should we keep a robots.txt file. and How to take away the strange url we see in all search engines as www.website.com/?C=S;O=A

This is shit and it is worrying and although searhed the web for a solutions couldn't found someone please help to take out this extension of ?C=S;O=A since my sourcecode is 100% clean & simple without any errors. Kindly assist in solving this issue since I want the website to be crawled openly like any other website with its sublinks as weel seen Ex. Type Daniweb and see it shows the website url clearly as wwww.daniweb.com and then all subpages links that a user can directly click and go. Help......

  • 3 Contributors
  • 2 Replies
  • 320 Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by cereal

Dani AI

Generated

As already indicated, the mysterious suffix is generated by the server’s directory‑listing/sorting machinery rather than by malicious code. Search engines index any URL that returns a normal HTML page (HTTP 200), so those auto‑generated directory pages with query parameters can end up in results. The practical goal is to stop the server from exposing indexable directory listings or to make those listings non‑indexable and canonicalize/redirect them to the clean URL.

Quick checks to run (inspect HTTP response and logs):

curl -I "http://www.website.com/?C=S;O=A"
grep 'C=S;O=A' /var/log/apache2/access.log

If the response is 200 and listings appear in the logs, fix it on the server side rather than relying solely on robots.txt.

Recommended remediation steps (in order of safety and SEO effectiveness):

  • Prevent directory listings from being served (server config or vhost .htaccess). If directory listings must remain for users, configure the server so it does not append sort query strings (see the autoindex settings in Apache documentation).
  • Add a server‑side 301 redirect that strips the query string for requests that match C=S;O=A. Example (mod_rewrite):
RewriteEngine On
RewriteCond %{QUERY_STRING} ^C=S;O=A$
RewriteRule ^ %{REQUEST_URI}? [R=301,L]
  • For already‑indexed URLs, return a noindex header for those requests so search engines remove them, e.g.:
SetEnvIf Query_String "^C=S;O=A$" noindex=1
Header set X-Robots-Tag "noindex, nofollow" env=noindex

After server fixes, request re‑indexing: submit an updated sitemap and use webmaster tools (Google Search Console / Bing Webmaster Tools) to remove or re‑fetch affected URLs. Robots.txt can help stop future crawling but is not a reliable de‑index mechanism on its own. These server changes remove the root cause so search results will show the clean domain and subpages instead of the auto‑generated query URLs.

Recommended Answers

All 2 Replies

Member Avatar for Member #120589

If you provide your url, perhaps we can have a look.

Member Avatar for cereal

This is created by the Apache Auto-Index module:

You can disable it by using Options -Indexes, or by placing an index.html (or index.php) into each directory. In alternative, if you need to create directory indexes, you should be able to stop the column sorting by using:

Options +Indexes
IndexOptions SuppressColumnSorting IgnoreClient

If you still have issues show your .htaccess file or the configuration of the virtual host.

Edited by cereal
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.