Question about gathering search results for a phrase with Python. I wrote a simple script for my personal SEO purposes that would go out and collect the top 30 search results on Google and then return a list of those URL's. The trouble is everytime I try to run my script Google starts returning an error about the search not being allowed. The URL I'm using works fine if entered manually but Google seems to know I'm trying to get access through a script. Is there a way around this or an API that I should be using? My searches into the subject haven't given me much.

9 Years
Discussion Span
Last Post by mn_kthompson

Google does not allow searches from any devices that does not provide a commonly used agent string. When you gather data using python the agent string is something like "python 2.6". Google doesn't allow that because they don't want people using scripts to gather their search results.

The answer is to spoof your agent string. I recently wrote up a script that does something similar to what you're trying to do. I wrote a blog entry about it and included sample code that you might want to check out.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.