Perl

Perl RSS

Memory Managment?

•

Join Date: Jan 2009

Posts: 106

Reputation: jen140 is an unknown quantity at this point

Solved Threads: 5

jen140

Offline

Junior Poster

Memory Managment?

Jul 31st, 2009

Hello all.
I have a small problem.
I have a crewler, that gets into one website and extract some info from it, and writes the exctracted content to file.
The only prob, that in 3~4hours of work, thise script uses 1 gig of ram.

 Help with Code Tags 
 Perl Syntax (Toggle Plain Text) 
for(my $increment = 1; $increment <= 999999; $increment++){
my $funky = "http://www.website.com/page?id=".$increment;
print $increment."\n";
my $content = get($funky);
for(my $increment = 1; $increment <= 999999; $increment++){
my $funky = "http://www.website.com/page?id=".$increment;
print $increment."\n";
my $content = get($funky);

THere is the part of the code, i tryied the undef function(at the end of for cycle) but the sam ram was used after 3~4hours of crewling.
So i would like to know, if is it possible to clean the memory at the end of the for cycle, and how .
Thanks in advance.

•

Join Date: May 2006

Posts: 66

Reputation: lordspace is an unknown quantity at this point

Solved Threads: 4

lordspace

Offline

Junior Poster in Training

Re: Memory Managment?

Jul 31st, 2009

Why don't you call an external binary e.g. curl to get the content.
You might want to add some headers (at least user agent) so the site doesn't block you by detecting the user agent string.
Executing an external binary will be a separate process and the memory will be released by the OS.

SEO Software for forums and CMS - SEO Filter | My Web Development Blog

•

Join Date: Jan 2009

Posts: 106

Reputation: jen140 is an unknown quantity at this point

Solved Threads: 5

jen140

Offline

Junior Poster

Re: Memory Managment?

Jul 31st, 2009

Ok ,thanks for a nice ideia, i am going to try it .But still , for the future would like to know how to manage memory in perl.

•

Join Date: Mar 2006

Posts: 898

Reputation: KevinADC has a spectacular aura about

Solved Threads: 67

KevinADC

Offline

Practically a Posting Shark

Re: Memory Managment?

Jul 31st, 2009

Normally in perl you don't worry about memory management. But instead of undef you can assign the variable an empty list/string and see if that helps free up memory.

$var = '';
@var = ();
%var = ();

•

Join Date: Jan 2009

Posts: 106

Reputation: jen140 is an unknown quantity at this point

Solved Threads: 5

jen140

Offline

Junior Poster

Re: Memory Managment?

Jul 31st, 2009

It helped a bit, at start . But in 20~30 minutes the memory used is same =( . Or possibly im missing some variable. Gonna check. Is there a way to print all the variable names ?