Member Avatar for Arun.N

i try to make a crawler that crawls a web page & retrieves
the stock information from google,but can't do it .
so plz help me 2 make that type of crawler.
urgent plz...

  • 6 Contributors
  • 5 Replies
  • 194 Views
  • 2 Years Discussion Span
  • Latest Post Latest Post by moobaa

Dani AI

Generated

For : the practical flow that resolves most of the thread's issues is (1) fetch the raw page, (2) confirm the HTML actually contains the data (not injected by JavaScript), (3) parse the HTML with a DOM parser, and (4) store snapshots for change-tracking. Several replies pointed to fetching ( and ) and to using feeds where possible ( ). Prefer an official data feed or API for stock data. Scraping should be a fallback.

A compact, robust parse step (example uses PHP DOM tools rather than regex) — load the fetched HTML into a DOM and query with XPath. This avoids brittle regex matching on changing markup.

<?php
libxml_use_internal_errors(true);
$html = /* fetched page as a string */;
$doc = new DOMDocument();
$doc->loadHTML($html);
$xpath = new DOMXPath($doc);
$node = $xpath->query("//span[contains(@class,'price')]")->item(0);
echo $node ? trim($node->textContent) : "not found";
?>

Common reasons for "Array()" or empty results (as reported): the fetch returned an error or redirect, the site requires cookies or JavaScript, the selector/regex no longer matches changed HTML, or the request was blocked. Debugging checklist: save the raw HTML to disk and open it in a browser, inspect the network response and headers, check HTTP status codes and any error messages, and try a browser automation tool when content is JS-rendered.

For change tracking ( ): normalize snapshots (remove timestamps/ads, collapse whitespace), store timestamped copies, then compute diffs with a text-diff library or server-side tool and present side-by-side. Always respect robots.txt and the target site's terms, add reasonable delays, and prefer authenticated API access when available.

Recommended Answers

All 5 Replies

Member Avatar for chrelad

Hi Arun.N,

Sounds like your looking for cURL... Have a look at the cURL documentation and see what you think.

cURL + regular expressions (preg_match_all) = exactly what your looking for.

I've written a few of these "crawlers" myself, so I'll include some foundational code for a very a simple one for you:

<?php

// Return a handle to a curl connection to the site you want to pull info from
$ch = curl_init('http://finance.google.com/finance');

// Set some options for the connection
curl_setopt($ch,CURLOPT_HEADER,0); // Don't return header information, although, this can be handy ;)
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1); // Give us the page source

// Open the connection with the options specified
$cr = curl_exec($ch);

// Run your regular expression against the source to pull what you want, you can use external programs to format the html for easier parsing if you want before you scan it.
preg_match_all('/href="()"/i',$cr,$pm,PREG_SET_ORDER);

// So you can see what you found
print_r($pm);

// Display the results again :D
foreach($pm as $pv) echo $pv[1] . "\r\n";

?>

Hope this helps!

Member Avatar for mario.stoica

<?php

$ch = curl_init("http://www.example.com/");
$fp = fopen("example_homepage.txt", "w");

curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);

curl_exec($ch);
curl_close($ch);
fclose($fp);
?>

Member Avatar for gopiyadav

Hi friends
I need some help from you guys.....I need a crawler such that it tracks the changes in the website content and it should show the track changes like oldcontent and newcontent should be shown side by side

Member Avatar for almostbob

use the rss feed of the stock page

Member Avatar for moobaa

Hi there...

I tried this cURL script, but all I get returned is "Array()"... I've got this running here:

Cheers,
Todd

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.