Member Avatar for ! !

Google Webaccelerator is being used to crawl and spider URLs

Interestingly enough, several brand new never before owned URLs have just been added to Google SERPs

What is so interesting is that they were purchased about a week ago - the hand coding of the site and the FTP-ing were only done a new computer that had the Google Accelerator on it.

They are NOT linked anywhere because they are still being worked on.

There have been URLs in the past without that Google Accerator Computer being used - purchased from the same domain reseller and using the same host - that were not added until someone began the process.

This is good if you want it added - but be forwarned - since Google now spiders Word, Powerpoint, Excel, Flash etc... be sure you want them on the Web :o

if you know what'a mean

But this also brings up an interesting point....
All the Visited URLs from everyone are in Google's main database - so they ALL are spidered - is this information being used in their ALGOs - not for JUST personalization - but in the General Algos (an equivalent to the Bookmark tracking sites - the more visits - the more Popular a site is) and will get a higher SERPs....hmmm...

  • 2 Contributors
  • 5 Replies
  • 194 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

Short primer and practical fixes that build on 's observation (and on 's test idea):

Google Web Accelerator behaved like a Google‑side proxy that prefetched links and cached pages; those background fetches were ordinary HTTP requests that could expose URLs and content to Google’s systems and—at least in beta—cause surprising visibility and privacy issues. (en.wikipedia.org)

How the accidental indexing happens and what to do immediately: GWA (and other prefetchers) scan pages for links and issue background GETs. Those requests often include special headers you can check for; blocking prefetch requests at the server is an effective short‑term defense. Example approaches you can drop into Apache/PHP:

# Apache (in .htaccess or vhost)
RewriteEngine on
RewriteCond %{HTTP_X_MOZ} prefetch
RewriteRule .* - [F]
/* PHP */
if (!empty($_SERVER['HTTP_X_MOZ']) && strpos($_SERVER['HTTP_X_MOZ'],'prefetch') !== false) {
  header('HTTP/1.1 403 Forbidden');
  exit;
}

This pattern (detect prefetch header → return 403) was suggested by early webapp authors and worked to stop accidental prefetch actions. (ianbicking.org)

Longer‑term / index control: don’t rely on robots.txt alone to hide in‑progress content. For non‑HTML files (Word, PPT, XLS, SWF, PDFs) return an HTTP X‑Robots‑Tag: noindex or add a <meta name="robots" content="noindex"> on pages you don’t want in Search; once Google sees the tag/header it will drop the URL from results. Also use Search Console’s URL removal and request recrawl after you fix the resource. Google documents both the noindex header and which file types it can index. (developers.google.com)

Practical checklist to follow now

  • Audit server logs for Google‑proxy requests and the HTTP_X_MOZ/prefetch header.
  • Add the prefetch‑block rule and a short noindex header for staging or private files.
  • Use Search Console removals and request recrawl after you apply noindex.
  • Treat all uploaded Office/Flash files as public unless explicitly protected—Google can and will index many non‑HTML formats.

Short note of caution: ’s case shows this can make brand‑new sites appear quickly, but don’t assume those fetches are a reliable ranking signal — they were a proxy/cache artifact and also raised obvious privacy concerns.

Recommended Answers

All 5 Replies

Member Avatar for Dani

Ya know what, I'll give it a go! :) I had a similar question about Personalized Searching not too long ago - I assume Google IS data mining this info in some way.

Member Avatar for ! !

not just THIS data - but toolbar data, click popularity, possibly Google WIFI, ......they have realized that the old days of just using LINK POPULARITY & keywords just won't go as this decade continues.

Everything that factors human behavior holds some potential info

Member Avatar for Dani

Yes, I definitely agree :)

Member Avatar for ! !

Here is something else to note about URLs

Google Search Quality Engineer / Representative did come out and admit that URLs with:


& % etc


are now excluded form counting as links - and are devalued

- He said avoid & % etc, "We wont look at them"

http://www.seroundtable.com/archives/002925.html

Member Avatar for Dani

But & is just a query string URL!!

(and please, no referring to any URLs in this forum because even if it's relevant, it becomes impossible to tell whether you're linking to an article on a site you own or are affiliated with)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.