how do I extract data from html file ?

[masterinex]

Hi,
I want write a program which
extract 'Rated PG for some scary moments and mild language' from the following html file and return it as a list .

html file:
<div class="info">
<h5><a href="/mpaa">MPAA</a>:</h5>

<div class="info-content">
Rated PG for some scary moments and mild language. (also 2009 extended version)
</div>
</div>

Why wouldnt this code work ?
mpaaget = re.compile('<h5><a href="/mpaa">MPAA</a>:</h5><div class="info-content">(.*?)</div>')
mpaa = mpaaget.findall(htmlr)

[ghostdog74]

use a html parser for this job, such as BeautifulSoup. If you don't want to, then another way is to read the whole html, split on "</div>", go through each element in the list, check for "<div class="info-content">", if found, replace it will null. You will get your string

[masterinex]

use a html parser for this job, such as BeautifulSoup. If you don't want to, then another way is to read the whole html, split on "</div>", go through each element in the list, check for "<div class="info-content">", if found, replace it will null. You will get your string

hey, I tried it with
mpaaget = re.compile('<div class="info-content">(.*?)</div>')
but then I got something else . Could it be because there is a new line after <div class="info-content"> ? How do I take care of that?

<div class="info-content">
Rated PG for some scary moments and mild language. (also 2009 extended version)
</div>

[masterinex]

ohw I got it now , thanks for pointing it out .

[jlm699]

hey, I tried it with
mpaaget = re.compile('<div class="info-content">(.*?)</div>')
but then I got something else . Could it be because there is a new line after <div class="info-content"> ? How do I take care of that?

Yes, the white space does not fit into your regular expression. Modify like so to match 0 or any number (*) of white space characters (\s):

>>> m = re.compile('<h5><a href="/mpaa">MPAA</a>:</h5>\s*<div class="info-content">\s*(.*?)\s*</div>')
>>> m.findall(h)
['Rated PG for some scary moments and mild language. (also 2009 extended version)']
>>> m.match(h)
>>>

[masterinex]

Yea , that was problem ,thanks for pointing it out again .
Looks like \n and \s* are the same character .

I have another question. lets say
I want to extract the number 7.2 from the html string below :

<a href="/ratings_explained">weighted average</a> vote of <a href="/List?ratings=7">7.2</a> / 10</p><p>

how come this doesnt work ?

averageget = re.compile('<a href="/List?ratings=7">(.*?)</a>')
average = averageget.findall(htmlr)

Could it be that there some special structures in the html file again which I missed out ?

[jlm699]

This time it's because '?' is a special character in regular expressions (you're using it inside your group). The question mark indicates a greedy match of 1 or more (where as the asterick (*) is a greedy match of 0 or more). To match the question mark character itself you need to escape it in your regex like so: \? . The full regular expression then becomes:

>>> c = re.compile('<a href="/List\?ratings=7">(.*?)</a>')
>>> c.findall(t)
['7.2']

[masterinex]

ohw I see so its the '?' that causing the trouble ,
what is t btw do I need to assighn a value to it ?

[ghostdog74]

if you want to use regex, you should compile your regex with re.DOTALL and re.M for multiline match.

[masterinex]

Im a little unfamiliar with Python , what are re.DOTALL and re.M are they modules ?

[vegaseat]

They are Flags for compile():
re.MULTILINE (or re.M) string and each line
re.DOTALL (or re.S) match any character, including a newline
re,IGNORECASE (or re.I) case-insensitive matching