Searching Webpages using Python

[Musing888]

The basic idea for this program is to for me to enter a code to a corresponding website (which is already in a dictionary in the body of the program) and search that webpage for a specific phrase and return a value. This is so that I can check number online constantly with little effort.

However, all of the sample programs online do no work for the website I am trying. Some of the times it works for only google.

1.

from urllib import urlopen
print urlopen('url').read()

2.

import shutil
import os
import time
import datetime
import math
import urllib

from array import array

filehandle = urllib.urlopen(url)

for lines in filehandle.readlines():
    print lines

filehandle.close()

3. webbrowser function

Whenver I use one of the above codes it returns the following:

The web browser you are using is not supported by this site.

Please download one of the following supported browsers.

<BR>

<BR>

<a href="http://www.microsoft.com/ie/download">http://www.microsoft.com/ie/download</a>

<BR>

<BR>

<a href="http://www.netscape.com/download">http://www.netscape.com/download</a>

<BR>

I have tried setting Internet Explorer as my default web browser and I received the same message. I am running Windows 7 x64 and appreciate any suggestions or ideas on other ways to accomplish my goal.

[griswolf]

Just a quick hint before I run off to another task.

From http://www.voidspace.org.uk/python/articles/urllib2.shtml

Headers

We'll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.

Some websites (like google for example) dislike being browsed by programs, or send different versions to different browsers [2] . By default urllib2 identifies itself as Python-urllib/x.y (where x and y are the major and minor version numbers of the Python release, e.g. Python-urllib/2.5), which may confuse the site, or just plain not work. The way a browser identifies itself is through the User-Agent header [3]. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer [4].

[Musing888]

Thanks you. I think that your suggestion might be exactly what I am looking for.

I will get back to you and tell you if it worked completely.