Member Avatar for varunsara

I fetch some data from the remote server and feed it into my site database .It having 9 lakhs of records(app) but the insertion of records stopped 60,180 records only.we use mailto function and exception handler to find the bugs but no response.Anyone please advice me whether the cron get timeout or we have error in code our code is

<?php
set_time_limit(0);
ignore_user_abort();
try
{
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "");
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);

// close cURL resource, and free up system resources
curl_close($ch);
}
catch (Exception $e) 
{
 echo 'Caught exception: ',  $e->getMessage(), "\n";
mail('varunxxxx@xxxxx.com', 'update', $message);    
}
?>
  • 3 Contributors
  • 4 Replies
  • 439 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by pritaeas

Dani AI

Generated

Quick summary and focused checks for this kind of “works for small sets, dies on big sets” problem.

Likely causes (beyond the undefined $message that found): silent cURL failures (curl_exec does not throw exceptions), server-side resource limits (PHP memory/execution, webserver/php-fpm timeouts, ulimits or OOM kills), database issues (deadlocks, large transactions, unique-key errors), or the remote endpoint throttling/aborting after many requests. Also, mail-based alerts are unreliable on many hosts; logs are better.

Concrete troubleshooting steps

  • Run the job from the CLI and capture all output so you see fatal errors:
    php -f /path/to/script.php > /tmp/cron.log 2>&1
  • Add permanent progress logging (timestamp, last processed ID, counter, memory_get_usage()), so you can resume and see exactly where it stops.
  • Check system logs: PHP error log, webserver/php-fpm logs, /var/log/syslog or /var/log/cron, and MySQL error log for aborted connections or lock errors.
  • Confirm cron vs web/CLI differences: CLI php.ini often differs (especially max_execution_time and memory_limit) and cron user may have stricter ulimit settings.

cURL and HTTP checks (use this pattern rather than relying on try/catch)

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 600);
$response = curl_exec($ch);
if ($response === false) {
  error_log('cURL error: '.curl_errno($ch).' '.curl_error($ch));
} else {
  $http = curl_getinfo($ch, CURLINFO_HTTP_CODE);
  error_log("HTTP $http after $processed records");
}
curl_close($ch);

Operational fixes to consider

  • Process in small batches (e.g. 1k–10k rows), commit/flush each batch and persist last ID so the job can resume.
  • Reduce transaction size and use prepared statements or bulk inserts (or LOAD DATA) to reduce DB load.
  • If the remote side is heavy, throttle calls or move the heavy work to a queue/worker (CLI) instead of a single long HTTP-triggered run.

Notes: logging beats mail for debugging long runs; use logs to capture the exact failure message and the last processed ID, then act on that clue.

Recommended Answers

All 4 Replies

Member Avatar for Member #120589

<off topic>

I'm assuming 9 lakh = 900,000

Stick to international amounts, lakh is a bit obscure.

Member Avatar for pritaeas

One thing I notice is $message is undefined (line 19). Am not so sure though, that curl raises exceptions on errors.

Member Avatar for varunsara

now i fix that $message = $e->getMessage(); There is no problem with the code . its working when am fetching minimum amount of data. when go for mass data its get terminated . I set infinite time for execution in web server but It seems to be a same issue. Kindly please advice and share your ideas

Edited by pritaeas because: Added markdown.
Member Avatar for pritaeas

I set infinite time for execution in web server

Are you sure your host actually supports this? For security/performance reasons this is usually disabled.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.