943,948 Members | Top Members by Rank

Web Development > PHP > Looking for PHP Web Spider Script
Ad:
  • PHP Discussion Thread
  • Unsolved
  • Views: 12101
  • PHP RSS
You are currently viewing page 1 of this multi-page discussion thread
Permalink
Sep 27th, 2008
0

Looking for PHP Web Spider Script

Expand Post »
Apologies first if this is covered elsewhere - I searched but could not find.

I am looking for a way to search the web for the presence of a JavaScript code snippet within the HTML <body> of a web page. I would specify the code snippet and send the bot on its way, it would come back with either a number of results, or a list of pages.

I realise there are billions of web pages so don't know whether this is feasilble or not.

The purpose is to determine the number of participating sites in a particular network. (Currently uncontrolled so any site could have the code on)

Any ideas on this?

Thanks
Similar Threads
Reputation Points: 10
Solved Threads: 0
Newbie Poster
monsterpot is offline Offline
1 posts
since Sep 2008
Permalink
Sep 27th, 2008
0

Re: Looking for PHP Web Spider Script

Click to Expand / Collapse  Quote originally posted by monsterpot ...
Apologies first if this is covered elsewhere - I searched but could not find.

I am looking for a way to search the web for the presence of a JavaScript code snippet within the HTML <body> of a web page. I would specify the code snippet and send the bot on its way, it would come back with either a number of results, or a list of pages.

I realise there are billions of web pages so don't know whether this is feasilble or not.

The purpose is to determine the number of participating sites in a particular network. (Currently uncontrolled so any site could have the code on)

Any ideas on this?

Thanks
I don't think you will find too many php web spider scripts only because php is an interpreted language and will be quite slow. I would look into Java probably.
Last edited by R0bb0b; Sep 27th, 2008 at 4:37 pm.
Reputation Points: 358
Solved Threads: 89
Posting Shark
R0bb0b is offline Offline
986 posts
since Jun 2008
Permalink
Sep 27th, 2008
0

Re: Looking for PHP Web Spider Script

Hi,

I agree with RobRob here, PHP just isn't powerful enough to be doing this sort of task. PHP is not a compiled language, so it is interpreted into machine-readable code on-the-fly. This is very slow in comparison to compiled languages such as C and C++ which naturally operate at a closer level to the hardware. Even if you had unlimited power and resources, PHP is very restrictive as it only runs once, and thanks to execution timeouts, you can't do very much.

If you're still interested in making a spider, I must say that it's no easy feat. You would be much better looking into C and C++ in subsidary forums, with most of you're focus spent on understanding networking.

Thanks,
Christopher Lord
Reputation Points: 10
Solved Threads: 1
Newbie Poster
heenix is offline Offline
15 posts
since Sep 2008
Permalink
Sep 28th, 2008
0

Re: Looking for PHP Web Spider Script

How about trying a Google search? You could search within the HTML for your Javascrpit code. If you would like it in script form, try using the Google Ajax Search widget and simply count the results!!!
Last edited by FlashCreations; Sep 28th, 2008 at 4:43 pm.
Reputation Points: 47
Solved Threads: 47
Posting Whiz
FlashCreations is offline Offline
393 posts
since Sep 2008
Permalink
Sep 29th, 2008
0

Re: Looking for PHP Web Spider Script

Quote ...
Quote ...
I don't think you will find too many php web spider scripts only because php is an interpreted language and will be quite slow.
Quote ...
I agree with RobRob here, PHP just isn't powerful enough to be doing this sort of task.
Well from the quotes above, I may just be able to prove all of that wrong with a PHP bot series I am working on. This post is not advertiesment but is to make the users 'R0bb0b' and 'heenix' and 'monsterpot' aware of what PHP is really capable of.
I have managed to make a bot which will index all of the urls that a linked to (within a tree type format) and will keep on indexing the web until it runs out of links. I will provide you with the source for you to make your javascript finder and currently the below script only indexes urls which is needed to scan more than one page.
php Syntax (Toggle Plain Text)
  1. if (isset($_GET['url']))
  2. 	{
  3. 	include("db.php");
  4. 	mysql_connect($dbhost,$accountname,$password)
  5. 	or die("Could not connect to MySQL server");
  6. 	mysql_select_db($database) or die(mysql_error()."Could not select database");
  7. 	$file=file_get_contents($_GET['url']);
  8. 	$links=preg_split('/(href\=\'|href\=\"|href\=)/is',$file);
  9. 	//$file=preg_replace('/(.*)(\ href\=\"|\ href\=\'|\ href=)(.*\ |.*\"|.*\'|.*>)/is','$2|^=^|',$file,1);
  10. 	//echo $file."<br>";
  11.  
  12. 	mysql_query("INSERT INTO `indextemp` SET `url`='".$_GET['url']."', `stage`='1'");
  13. 	$id=1;
  14. 	while (isset($links[$id]))
  15. 		{
  16. 		$links[$id]=preg_replace("/([^\'])\'(.*)/is",'$1',$links[$id]);
  17. 		$links[$id]=preg_replace("/([^\"])\"(.*)/is",'$1',$links[$id]);
  18. 		$links[$id]=preg_replace("/([^\>])\>(.*)/is",'$1',$links[$id]);
  19. 		$links[$id]=preg_replace("/([^ ])\ (.*)/is",'$1',$links[$id]);
  20. 		$links[$id]=preg_replace("/([^\'])\'(.*)/i",'$1',$links[$id]);
  21. 		$links[$id]=preg_replace("/([^\"])\"(.*)/i",'$1',$links[$id]);
  22. 		$links[$id]=preg_replace("/([^\>])\>(.*)/i",'$1',$links[$id]);
  23. 		$links[$id]=@preg_replace("/([^ ])\ (.*)/i",'$1',$links[$id]);
  24.  
  25. 		$ifexists=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  26. 		if (mysql_num_rows($ifexists)==0 && strlen($links[$id])>16)
  27. 			{
  28. 			mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  29. 			echo $links[$id]."<br>";
  30. 			}
  31. 		$id+=1;
  32. 		}
  33. 	unset ($links);
  34. 	$continue=1;
  35. 	while ($continue=1)
  36. 		{
  37.  
  38. 		$sqllinksa=mysql_query("SELECT * FROM `indextemp` WHERE `stage`='0'");
  39. 		while ($sqllinks=mysql_fetch_array($sqllinksa))
  40. 			{
  41. 			$file=file_get_contents($sqllinks['url']);
  42. 			$links=preg_split('/(href\=\'|href\=\"|href\=)/is',$file);
  43.  
  44. 			mysql_query("UPDATE `indextemp` SET `stage`='1' WHERE `url`='".$sqllinks['url']."'");
  45. 			$id=1;
  46. 			while (isset($links[$id]))
  47. 				{
  48. 				$links[$id]=preg_replace("/([^\'])\'(.*)/is",'$1',$links[$id]);
  49. 				$links[$id]=preg_replace("/([^\"])\"(.*)/is",'$1',$links[$id]);
  50. 				$links[$id]=preg_replace("/([^\>])\>(.*)/is",'$1',$links[$id]);
  51. 				$links[$id]=preg_replace("/([^ ])\ (.*)/is",'$1',$links[$id]);
  52. 				$links[$id]=preg_replace("/([^\'])\'(.*)/i",'$1',$links[$id]);
  53. 				$links[$id]=preg_replace("/([^\"])\"(.*)/i",'$1',$links[$id]);
  54. 				$links[$id]=preg_replace("/([^\>])\>(.*)/i",'$1',$links[$id]);
  55. 				$links[$id]=preg_replace("/([^ ])\ (.*)/i",'$1',$links[$id]);
  56.  
  57. 				$ifexist=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  58. 				if (strlen($links[$id])>5 && mysql_num_rows($ifexist)==0)
  59. 					{
  60. 					mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  61. 					echo $links[$id]."<br>";
  62. 					} else { unset($ifexists['url']); }
  63. 				$id+=1;
  64. 				}
  65.  
  66.  
  67. 			$ifexists=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  68. 			$ifexists=mysql_fetch_array($ifexists);
  69. 			if (!isset($ifexists['url']) && strlen($links[$id])>5)
  70. 				{
  71. 				mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  72. 				echo $links[$id]."<br>";
  73. 				}
  74. 			}
  75. 		$checkcontinue=mysql_query("SELECT * FROM `indextemp` WHERE `stage`='0'");
  76. 		if (mysql_num_rows($checkcontinue)==0)
  77. 			{
  78. 			$continue=0;
  79. 			break;
  80. 			}
  81. 		}
  82. 		//
  83. 	}
  84. echo "<form><input type='text' name='url' size=50><input type='submit' value='index'></form>";
  85. ?>

Below is a second file named 'db.php' with sql configurations.
php Syntax (Toggle Plain Text)
  1. <?
  2. $accountname='root';
  3. $password='';
  4. $dbhost='localhost';
  5. $database='mydatabasename';
  6. ?>
Also, in the mysql database, there is a table named 'tempindex' with the columns 'url' and 'stage'. (but remove all the quotes in the previous sentence.)

Comment to monsterpot:
If you are still interested in making a PHP bot to find your javascript on the web then just let me know as I can help you there. Also I would need to know exactly what that javascript code is and what parts of that javascript code can change.
Sponsor
Featured Poster
Reputation Points: 410
Solved Threads: 258
Occupation: Genius
cwarn23 is offline Offline
3,004 posts
since Sep 2007
Permalink
Sep 29th, 2008
0

Re: Looking for PHP Web Spider Script

Click to Expand / Collapse  Quote originally posted by cwarn23 ...
Well from the quotes above, I may just be able to prove all of that wrong with a PHP bot series I am working on. This post is not advertiesment but is to make the users 'R0bb0b' and 'heenix' and 'monsterpot' aware of what PHP is really capable of.
I have managed to make a bot which will index all of the urls that a linked to (within a tree type format) and will keep on indexing the web until it runs out of links. I will provide you with the source for you to make your javascript finder and currently the below script only indexes urls which is needed to scan more than one page.
Sure, nobody said it couldn't be done, it'll just take all day, possibly all week depending on how many sites you plan to spider. Google actually claims to index 2e9 pages. Let's say you get a fourth that at some point(500,000,000), and let me be generous and say that each page has an average of 10 links per page. If I am reading your script right that would be 5,000,000,000 loops. Last time I clocked PHP it was running about 600000 loops per minute on a 3g processor with 1g ram. That would average just over 833 minutes given that you had absolutely no memory leaks while the script is running and this doesn't include the file_get_contents() and mysql_query() functions that you will be running. I personally wouldn't do it with php because PHP is not a multi-threading language(which would be extremely helpful in this case) and PHP is several times slower than C++.

But for a side project, maybe.
Last edited by R0bb0b; Sep 29th, 2008 at 1:34 am.
Reputation Points: 358
Solved Threads: 89
Posting Shark
R0bb0b is offline Offline
986 posts
since Jun 2008
Permalink
Sep 29th, 2008
0

Re: Looking for PHP Web Spider Script

Quote ...
Sure, nobody said it couldn't be done, it'll just take all day, possibly all week depending on how many sites you plan to spider. Google actually claims to index 2e9 pages.
Just to add to the statistics, I remember reading in Google 'to the power of project' that Google now claims to have 1 trillion pages indexed. Also from the Google web alerts I have
(eg. inurl:cwarn23.info) I have found that Google usually revisits the same website within 8 days but one time in the past year has taken 14 days. Also I have tested my script to index an average of 16 pages per second (at the most). So I hope you find those statistics useful.
Sponsor
Featured Poster
Reputation Points: 410
Solved Threads: 258
Occupation: Genius
cwarn23 is offline Offline
3,004 posts
since Sep 2007
Permalink
Dec 30th, 2008
0

Re: Looking for PHP Web Spider Script

The stupidity in this thread made me register an account. Originally I was googling for the same thing the original poster is looking for.

When you crawl the web, you will spend most time waiting for network packets and saving the data some place. A spider is a perfect example of a piece of code, where execution time does not matter at all. You could write it in commodore basic 2.0 and wouldn't notice a difference.

Creating a spider in an unsuitable language like C++ will double your development effort for an actual performance gain in the first percentile.

The execution time limit in PHP is actually configurable. (Doh). It's usually disabled for command line execution. PHP only runs once? What happens then? The script self destructs?

Recommending java over PHP for performance reasons only makes sense if you are religious and worship The Java.

Naturally running a web crawler is not a task for days or even weeks. It's closer to years. If the op was looking to crawl a single site or two, he'd probably use one of the perfectly fine windows client applications and not look for a script.

There are opcode caches for PHP, which make it a just-in-time compiled language.

Assuming that there are no memory leaks is quite generous. Can we also assume that the world is round? When you "clock" a programming language, it would be kinda helpful to know what that loop was running, which operating system you were using, the bus width, and the compiler flags for the executable. The amount of memory does rather not matter.

Oh, and ...

PHP Syntax (Toggle Plain Text)
  1. <?php
  2. $count = 0;
  3. $now = microtime(true);
  4. while ( ($now+1) > microtime(true)) $count++;
  5. print "Loops per second: ".number_format($count)."\n";
  6. ?>
  7.  
  8. workhorse:~# php loop.php
  9. Loops per second: 2,222,026

... you are full of it.

But let's assume you actually benchmarked the script in question. Let's also assume an average text weight of 50kb for a web page. Then your 3g processor (mobile phone?) could spider 30 gigabyte per minute. That's ~500 megabytes per second. Phat subsystem there. Mysql cluster with memory tables on 10GbE?

There are simple ways to split the websites to crawl between several instances of the script. You do not need threads. You can multi-task.

You remind me of that dude who threatened to "hack my website" and backed that claim with a traceroute. Please stop giving technical advice. Thank you.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
Mantriur is offline Offline
2 posts
since Dec 2008
Permalink
Dec 30th, 2008
0

Re: Looking for PHP Web Spider Script

But to answer the original question:

http://vision-media.ca/resources/php...aper-5-minutes

Just modify one of the regexp to identify the script tags you are looking for and use the get_links function to identify the next target.

Dump the links into a mysql table together with a flag if/when crawled and feed the spider from that table. Dump positives into another table with the URL. You can then query the results from that one any time.

If you run this on an external server or webspace, observe your bandwidth/volume limits. The internet is big.
Last edited by Mantriur; Dec 30th, 2008 at 9:33 am.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
Mantriur is offline Offline
2 posts
since Dec 2008
Permalink
Jul 14th, 2009
0

Re: Looking for PHP Web Spider Script

I had try the script..did it just crawl in homepage..And it crawl external link to like google and ect..

How to make it just crawl the internal link in all pages, and it never crawl the same internal link..Please tell me how to do that..? Urgent
Reputation Points: 10
Solved Threads: 0
Light Poster
crohole is offline Offline
34 posts
since Jul 2009

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in PHP Forum Timeline: how to go back to a php page keeping the search criteria?
Next Thread in PHP Forum Timeline: Foreach Loop Error (Duplicates Last Entry)





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC