Member Avatar for Vandithar

Hi,

I have a URL like this:

$url="http://www.somesite.com";

I have to connect to this URL and download the data. Sometimes there will be problem in that particular site. So I have to reconnect and then again download the data.

Here is the code to connect online.

#!/usr/bin/perl 

use CGI;
use LWP;
use DBI;
use CGI qw(:standard);
use LWP::UserAgent;

$fetch="http://www.somesite.com";

$ua=new LWP::UserAgent;
$res=$ua->get($fetch);
$result=$res->content;
print "<br>$result<br>";

How can i write a program such that if a data can't be downloaded at first time suppose if any problem persists how can i reconnect to that perticular site and download the data ?

If it can try connecting 5 times and downloading the data and if 5th time also it fails then the program should stop try connecting automatically.

How can i do this?

Any ideas??

Regards
Vanditha

  • 3 Contributors
  • 4 Replies
  • 216 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by KevinADC

Dani AI

Generated

Short answer: use a controlled retry loop with a reasonable timeout, a pause (preferably exponential backoff with jitter) between attempts, and only retry on transient failures (network errors, 5xx responses, 408/429). Log the status_line each try so you have a record of why an attempt failed.

Good starting points already in the thread: correctly suggested checking the response, and suggested retrying in a loop. Two small pitfalls to watch for in the existing replies: a misspelled variable ($sucess in ’s snippet) and numeric comparisons done with string operators (eq in ’s snippet). Also avoid busy tight-loop retries or GOTO-based loops; use lexical variables, use strict/use warnings, and add a timeout on the user agent so a stalled request won’t hang everything.

A concise, practical pattern (example) that addresses those points:

use strict;
use warnings;
use LWP::UserAgent;
use Time::HiRes qw(sleep);

my $ua = LWP::UserAgent->new(timeout => 10, agent => 'Fetcher/0.1');
my $url = 'http://www.somesite.com';
my $max = 5;
my $backoff = 1;

for my $try (1..$max) {
    my $res = $ua->get($url);
    if ($res->is_success) {
        print $res->decoded_content;    # text; use content for binary
        last;
    }

    warn "Attempt $try: " . $res->status_line . "\n";

    my $code = $res->code || 0;
    last if ($code >= 400 && $code < 500 && $code != 408 && $code != 429); # don't retry permanent client errors

    last if $try == $max;
    sleep($backoff + rand(0.5));   # jitter
    $backoff *= 2;
}

Notes and troubleshooting: set the UA timeout to avoid long hangs; prefer decoded_content for text to respect charset; retry for 5xx, 408, 429 but stop on 404/401/403; add logging/alerts if all attempts fail. For large or resumable downloads look into HTTP Range support or streaming to disk rather than buffering the entire response.

Recommended Answers

All 4 Replies

Member Avatar for wickedxter

add this

if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

this was taken from the cpan site on this module.

Member Avatar for Vandithar

add this

if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

this was taken from the cpan site on this module.

Hi,

Thanks for the reply.

I have one doubt how can i check for 5 times using the above code i.e i will try reconnecting 4 times if i try to reconnect 5th time it should display saying that "Can't try now!!! try little later".

How can i do this???

Any ideas???

Regards
Archana

Member Avatar for wickedxter 
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $counters = {start => 0, end =>5};

REQUESTS:
my $req = $ua->get( '');

if ($req->is_success) {
      print $req->content;
  }
  else {
        $counters->{start}++ if $counters->{start} <= $counters->{end}; 
       print "Failed to retrive site, try again later." if $counters->{start} eq $counters->{end};

GOTO REQUESTS if $counters->{start} <= $counters->{end};
}

this is off the top of my head, you may need to tinker with it to get it work.

Member Avatar for KevinADC

or using a loop:

use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $success = 1;
for (1..5) {
   my $req = $ua->get( '');
   if ($req->is_success) {
      print $req->content;
      $sucess = 0;   
      last;
  }
}
unless ($sucess) {
    print "Failed all 5 tries to get content\n";
}
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.