Perl

Perl RSS

Generalized Web Scraping

•

Join Date: May 2009

Posts: 2

Reputation: hopewemakeit is an unknown quantity at this point

Solved Threads: 0

hopewemakeit hopewemakeit is offline

Offline

Newbie Poster

Generalized Web Scraping

May 28th, 2009

I am an undergraduate Student, from Computer Science and engineering department

I can construct a crawler in Perl, for one particular web-site to fetch the useful information, in my case the - Job Ads at that company's webpage.

Now, I want to construct some crawler that is generalized for say around 100 companies, using Perl

How can I do it ? I need some ideas/code/resource... and Do I need to study all 100 HTML codes?

Regards,
Kunal

•

Join Date: Feb 2009

Posts: 57

Reputation: mitchems is an unknown quantity at this point

Solved Threads: 2

mitchems

Offline

Junior Poster in Training

Re: Generalized Web Scraping

Jun 5th, 2009

Look into

Help with Code Tags

Perl Syntax (Toggle Plain Text)

HTML::Parse

. It is event driven and tag driven. You create functions when a tag opens or closes and how to deal with it. Play around with it and see if you can more effectively parse HTML with it.

And don't tell me there isn't one bit of difference between null and space, because that's exactly how much difference there is.

Larry Wall

•

Join Date: Feb 2009

Posts: 57

Reputation: mitchems is an unknown quantity at this point

Solved Threads: 2

mitchems

Offline

Junior Poster in Training

Re: Generalized Web Scraping

Jun 5th, 2009

I apologize in advance for back-posting. I meant

Help with Code Tags

Perl Syntax (Toggle Plain Text)

HTML::Parser
 
HTML::Parse is deprecated

And don't tell me there isn't one bit of difference between null and space, because that's exactly how much difference there is.

Larry Wall

•

Join Date: May 2009

Posts: 2

Reputation: hopewemakeit is an unknown quantity at this point

Solved Threads: 0

hopewemakeit hopewemakeit is offline

Offline

Newbie Poster

Re: Generalized Web Scraping

Jun 12th, 2009

hi,
thanks for the post

I have made crawlers for one web-site and it really is based on the Job-portal on that site and its HTML coding.. as in , like for what HTML tag opens and closes, and accordingly the data retrieval.,in between them (the one i need)

But I really cant figure it out, there are 100 web pages before me and I need to create a common scraper and all the HTML codes/tags are different.