Of course it's the case. The entire Adwords/Adsense system is profoundly flawed. It's like Las Vegas, where the casinos always win.

Google scrapes the web for content, caches it locally on their servers without permission, and then sells keywords against that content (which they didn't generate).

The initial idea of PageRank was very good: that a site's popularity could be measured through backlinks, which can be considered as "votes". Back when the web was new, that was valid.

As soon as they started selling keywords, monetizing content, and leaking little tidbits about their practices to fuel the SEO craze, they lost all credibility, in my opinion.

Sorry, I fail to see the unethical aspect to search engine indexing and crawling. The way that I see it, Google provides you with a service. They send you free traffic. You have the option of not receiving this free traffic by using a robots.txt file. However, if you enjoy the service Google offers you, which is sending visitors to your site, then you shouldn't complain about them indexing your content.

Sorry, it was just the way you said Google "caches [your content] locally on their servers without permission."

There is a difference between indexing, and caching. There is also a difference between behaving as a legitimate search engine, vs. scraping the entire web for content, and then re-packaging that content as sales goods.

Google is like a virus that co-opts some mechanism in the organism to reproduce. Because of Adsense/Adwords, the web "content" these days is turning into pre-packaged, easy-to-digest, monetized content for Google. Hardly anyone puts out a site these days without excessive hand-wringing about SEO/Google: what keywords to pick, will using SiteMap or Analytics make them better indexed, and similar nonsense.

There was even a question on the forum here, about changing servers: "Will having a new IP address hurt my rankings?"

This is the new state of the web.

And yet ... you don't use a robots.txt file to ban googlebot on your site.

I certainly use robots.txt. I don't prevent googlebot, you're right, because it isn't clear to me that Google differentiates between caching and indexing. Their FAQ on the subject of caching contains instructions for removing a site from their "index". They are being purposely vague about the differences.

If there is an instruction I can add to robots.txt that will prevent caching but allow indexing, I'd certainly like to know what it is.

Indexing, according to Google, is storing a local version of your pages within their database, to be searched. Their search engine works by searching a local version of your pages. How else would you like them to be able to perform fast searches if the content being searched wasn't local to them? Confused where the ambiguity is.

A book, and a book's index, are separate things, Dani. If Google wants to index my site, and return my site's URL in response to a search on indexed topics (digital printing, PostScript, PDF, etc.) then you're right: this is a service to me and my site. Thanks.

Google is the only search engine that uses a complete, cached copy, of your pages, and then uses their local cache, to generate an index.

cache != index

Even that would be OK, but they then allow users to view the cache, rather than going to your page.

Moreover, they allow companies to bid on keywords, which completely bypasses the natural search engine mechanism.

The combination of caching content (from Web, from Gmail, from Print, and now with Google Apps, add in all your business documents, and with Google checkout, your user's buying patterns), plus monetizing that content, is the problem.

A book and book's index are separate things, yes. But what I am saying is that Google's index and Google's cached version of pages are synonymous.

Allowing users to view the cache ... ok, I'll give you that you may not like this "feature". Perhaps it shouldn't exist. Fine. But if you can appreciate the service Google offers you in sending free traffic, you have to take the good with the bad.

Google's variation of keyword bidding is 1000% better than what some other search engines, such as the old goto.com, used to do. Google, at least, clearly labels sponsored listings in a separate area than natural search results. Goto.com used to intermingle them all together, so every search was just a list of sponsored links.

But, like I said, if you aren't comfortable with Google's caching / indexing / whatever you call it, then don't use them. But if you are thankful for the service they provide to you, don't complain about the negatives, which are far outweighed by the amount of traffic they are able to give us website owners BECAUSE of the way they spider content and not in spite of it.

We definitely have evaluated the risk/reward ratio differently here. I see the RISK of privacy abuses by Google as a result of them caching everything they can get their hands on (not just the web, but emails, books, business documents, site traffic stats, buying habits, search terms), and then re-selling it for profit, far outweighs the rewards.

I don't know for certain (because they ain't saying), but I would be very surprised if what you say is true, that their cache IS their index. I've asked, and have had no reply. You're saying they search their entire cache in response to each and every web query? I really doubt it.

I'll ask again, does anyone know? If I add a 'no-cache' directive to my robots.txt, will my site still be indexed? If not, you're absolutely right: I'll have to decide if it's worth it to me, or not, to be indexed by Google.