Member Avatar for wawammm

Hi All,

i'm having difficulties downloading about 4000 images from my php script from individual http locations.

It works as follows, every object is an xml file, and each xml file has an unknown number of attachments (images)

i loop through the xml files and then through the attachments where i use file_put_contentsto save the image to my server.

the problem is that i do get the images to my server, but strangely a lot of images get corrupt, it's always the first image of the next xml file.

any thoughts here guys?

Gr Jos

  • 2 Contributors
  • 3 Replies
  • 237 Views
  • 1 Hour Discussion Span
  • Latest Post Latest Post by cereal

Dani AI

Generated

Quick summary and likely causes: the pattern you describe (the first image of the next XML file is frequently corrupted) usually points to truncated downloads, wrong HTTP responses (HTML error pages saved as .jpg), filename collisions, or the remote host throttling/closing connections when hammered. is already on the right track; 's timeout suggestion and checking logs are good first steps. The following adds concrete checks and a more robust download pattern to try.

First, diagnostic checklist to run before large runs

  • Try downloading a few of the failing URLs by hand (curl or a browser) and compare the saved file to what your script produces.
  • Log URL, HTTP status, Content-Type and Content-Length for each download. If Content-Type is not an image or Content-Length is much larger than your saved file, the download was truncated or redirected to an error page.
  • After saving, validate the file with getimagesize() or finfo_file(); if these fail, delete and retry.
  • Make sure filenames are unique (ID + index may collide across feeds) and the PHP process has proper write permissions.

A robust, memory-friendly download pattern (use cURL and write straight to disk):

function download_image_to($url, $dest) {
    $fp = fopen($dest, 'wb');
    if (!$fp) return ['ok'=>false,'error'=>'open_failed'];
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_FILE => $fp,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_FAILONERROR => true,
        CURLOPT_CONNECTTIMEOUT => 10,
        CURLOPT_TIMEOUT => 30,
        CURLOPT_BUFFERSIZE => 8192,
        CURLOPT_USERAGENT => 'Mozilla/5.0'
    ]);
    $ok = curl_exec($ch);
    $err = curl_error($ch);
    $http = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);
    fclose($fp);
    if (!$ok || $http !== 200) { @unlink($dest); return ['ok'=>false,'error'=>$err,'http'=>$http]; }
    if (getimagesize($dest) === false) { @unlink($dest); return ['ok'=>false,'error'=>'not_image']; }
    return ['ok'=>true];
}

Operational tips

  • Run the script from the CLI (no max_execution_time limits) and use short sleeps (usleep(100000)) if the remote server is sensitive.
  • Retry 2–3 times with exponential backoff for transient errors.
  • If you need speed, use curl_multi but cap concurrency (4–8) to avoid remote throttling.
  • Add logging for URL, HTTP code, bytes written and a small hash (md5_file) so you can quickly find which entries failed.

If issues persist after these checks, capture a raw failing response (save the first 1–2 KB of the file and the HTTP headers) and compare it — that will quickly show whether the server is returning an error page or the stream is being cut off.

Recommended Answers

All 3 Replies

Member Avatar for cereal

So what happens, in your code, when you switch between the xml files? Can you show the code?

Member Avatar for wawammm

Hi Thanx, sure here it is

$xml_files = JFolder::files($searchpath, '.xml');
$count = count($xml_files);
$counter=0;
foreach($xml_files as $xml_file)
{

    $xml = simplexml_load_file(JPATH_COMPONENT_SITE.'/properties/'.date('d-m-Y').'/'.$xml_file);    

foreach($xml->RealEstateProperty->Attachments->Attachment as $attachment)
{

#DB INSERT HERE

file_put_contents(JPATH_COMPONENT_SITE.'/properties/assets/'.$xml->RealEstateProperty->PropertyInfo->ID.'_'.$attachment->Index.'.jpg', fopen($attachment->URLNormalizedFile, 'r'));


}   

$counter++;

}
Member Avatar for cereal

It could be a timeout from remote. The default value for timeouts is defined in php.ini file, but you could create a stream context to test the download:

$options["http"] = array(
        "method"    => "GET",
        "timeout"   => 15, # seconds
    );

fopen($attachment->URLNormalizedFile, "r", FALSE, $context);

Also check access and error logs from server and PHP.

but strangely a lot of images get corrupt, it's always the first image of the next xml file

Have you tried to manually access to the first entries of these XML files? Are these images reachable, have you tried a simple script to download just these files? Have you tried to open one of these files through an editor? Sometimes you expect binary but you find an HTML page (404 error pages for example).

By the way: why are you using fopen to get data into file_put_contents?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.