Ad:

Python Discussion Thread
Marked Solved
Views: 753

Page 1
2
Next

You are currently viewing page 1 of this multi-page discussion thread

Sep 21st, 2009

urllib2 problem

Expand Post »

Hi, I have this code:

python Syntax (Toggle Plain Text)
import urllib2 as url
import webbrowser
 
def extract(text, sub1, sub2):
    """
    extract a substring from text between first
    occurances of substrings sub1 and sub2
    """
    return text.split(sub1, 1)[-1].split(sub2, 1)[0]
start="http://xkcd.com/"
permlist=[]
textlist=[]
for i in range(1, 638):
    temp=start+str(i)
    permlist.append(str(url.urlopen(temp).readlines()[88]))
    textlist.append(str(url.urlopen(temp).readlines()[77]))
 
for i in permlist:
    i = extract(i, '<h3>Permanent link to this comic: ', '</h3>')
 
for i in textlist:
    i = extract(i, '<img src="http://imgs.xkcd.com/comics/scribblenauts.png" title="', '"')
 
 
print zip(permlist, textlist)
import urllib2 as url
import webbrowser

def extract(text, sub1, sub2):
    """
    extract a substring from text between first
    occurances of substrings sub1 and sub2
    """
    return text.split(sub1, 1)[-1].split(sub2, 1)[0]
start="http://xkcd.com/"
permlist=[]
textlist=[]
for i in range(1, 638):
    temp=start+str(i)
    permlist.append(str(url.urlopen(temp).readlines()[88]))
    textlist.append(str(url.urlopen(temp).readlines()[77]))

for i in permlist:
    i = extract(i, '<h3>Permanent link to this comic: ', '</h3>')

for i in textlist:
    i = extract(i, '<img src="http://imgs.xkcd.com/comics/scribblenauts.png" title="', '"')


print zip(permlist, textlist)

and whenever I run it, it raises this error:

Python Syntax (Toggle Plain Text)
Traceback (most recent call last):
  File "C:/Python26/test.py", line 15, in <module>
    permlist.append(str(url.urlopen(temp).readlines()[88]))
  File "C:\Python26\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python26\lib\urllib2.py", line 389, in open
    response = meth(req, response)
  File "C:\Python26\lib\urllib2.py", line 502, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python26\lib\urllib2.py", line 427, in error
    return self._call_chain(*args)
  File "C:\Python26\lib\urllib2.py", line 361, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 510, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "C:/Python26/test.py", line 15, in <module>
    permlist.append(str(url.urlopen(temp).readlines()[88]))
  File "C:\Python26\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python26\lib\urllib2.py", line 389, in open
    response = meth(req, response)
  File "C:\Python26\lib\urllib2.py", line 502, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python26\lib\urllib2.py", line 427, in error
    return self._call_chain(*args)
  File "C:\Python26\lib\urllib2.py", line 361, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 510, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found

What is the problem, but mainly what can I do to fix it?

thanks in advance

Similar Threads

Searching Weather for a term with urllib2 module in Python
I had problem with onmouse out in div in JavaScript / DHTML / AJAX
urllib2 fetching different page than browser in Python
HTML table parse problem (beautiful soup) in Python
MySQLdb interface problem in MySQL

leegeorg07

Reputation Points: 35

Solved Threads: 32

Posting Pro in Training

Offline

428 posts
since Jul 2008

Permalink

Sep 21st, 2009

Re: urllib2 problem

Looks like one of the 638 web pages is not available. You should use a try/except trap for this case.

sneekula

Reputation Points: 961

Solved Threads: 211

Nearly a Posting Maven

Offline

2,413 posts
since Oct 2006

Permalink

Sep 21st, 2009

Re: urllib2 problem

so what could I use?

sorry, at the moment I just want a quick fix and will figure out the best way when I have time

leegeorg07

Reputation Points: 35

Solved Threads: 32

Posting Pro in Training

Offline

428 posts
since Jul 2008

Permalink

Sep 21st, 2009

Re: urllib2 problem

python Syntax (Toggle Plain Text)
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))
        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except Error: # catch any exception and continue the for loop
        print "Error at index %d."%i
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))
        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except Error: # catch any exception and continue the for loop
        print "Error at index %d."%i

djidjadji

Reputation Points: 38

Solved Threads: 18

Light Poster

Offline

28 posts
since Aug 2009

Permalink

Sep 22nd, 2009

Re: urllib2 problem

Yeah you'll need to use exceptions, but if you want the script to continue after the error you're going to have to "pass" it, try this:

Python Syntax (Toggle Plain Text)
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except Error, err:
        print "Index Error: %d at %d" % (err, i)
        pass
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except Error, err:
        print "Index Error: %d at %d" % (err, i)
        pass

this will not only print the error and the location of the error but will also pass to keep the loop going.

ov3rcl0ck

Reputation Points: 35

Solved Threads: 22

Junior Poster

Offline

113 posts
since Sep 2009

Permalink

Sep 22nd, 2009

Re: urllib2 problem

hey again, they are good ideas but whenever I try to run it again it says:

Python Syntax (Toggle Plain Text)
Traceback (most recent call last):
  File "C:\Python26\test.py", line 18, in <module>
    except Error, err:
NameError: name 'Error' is not defined
Traceback (most recent call last):
  File "C:\Python26\test.py", line 18, in <module>
    except Error, err:
NameError: name 'Error' is not defined

leegeorg07

Reputation Points: 35

Solved Threads: 32

Posting Pro in Training

Offline

428 posts
since Jul 2008

Permalink

Sep 22nd, 2009

Re: urllib2 problem

Since you don't know the specific error class, simply use ...

python Syntax (Toggle Plain Text)
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))
        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except: # catch any exception and continue the for loop
        print "Error at index %d."%i
        pass
for i in range(1, 638):
    try:
        temp=start+str(i)
        permlist.append(str(url.urlopen(temp).readlines()[88]))
        textlist.append(str(url.urlopen(temp).readlines()[77]))
    except: # catch any exception and continue the for loop
        print "Error at index %d."%i
        pass

vegaseat

Moderator

Reputation Points: 1333

Solved Threads: 1403

DaniWeb's Hypocrite

Offline

5,792 posts
since Oct 2004

Permalink

Sep 22nd, 2009

Re: urllib2 problem

ok thanks, trying it now, So that I can do better handling soon, how can I find the class?

leegeorg07

Reputation Points: 35

Solved Threads: 32

Posting Pro in Training

Offline

428 posts
since Jul 2008

Permalink

Sep 22nd, 2009

Re: urllib2 problem

Well you found it in your first post ...
HTTPError

vegaseat

Moderator

Reputation Points: 1333

Solved Threads: 1403

DaniWeb's Hypocrite

Offline

5,792 posts
since Oct 2004

Permalink

Sep 23rd, 2009

Re: urllib2 problem

Oh ok thanks, whenever I run the zip part it uses the original text, not what I changed it to, I tried:

Python Syntax (Toggle Plain Text)
for i, j in permlist, textlist:
  print i, ':', j
for i, j in permlist, textlist:
  print i, ':', j

but it says that it is out of range, what can I do? I have googled it to no avail

leegeorg07

Reputation Points: 35

Solved Threads: 32

Posting Pro in Training

Offline

428 posts
since Jul 2008

Page 1
2
Next

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Python Forum Timeline: Open directory dialog box

Next Thread in Python Forum Timeline: Caeser cipher code breaker help (homework)

DaniWeb Message

Cancel Changes

User Name
Password