Looking for PHP Web Spider Script

Reply
1 2

•
•
Join Date: Sep 2008
Posts: 1
Reputation: monsterpot is an unknown quantity at this point 
Solved Threads: 0
monsterpot monsterpot is offline Offline
Newbie Poster

Looking for PHP Web Spider Script

 
0
  #1
Sep 27th, 2008
Apologies first if this is covered elsewhere - I searched but could not find.

I am looking for a way to search the web for the presence of a JavaScript code snippet within the HTML <body> of a web page. I would specify the code snippet and send the bot on its way, it would come back with either a number of results, or a list of pages.

I realise there are billions of web pages so don't know whether this is feasilble or not.

The purpose is to determine the number of participating sites in a particular network. (Currently uncontrolled so any site could have the code on)

Any ideas on this?

Thanks
Reply With Quote Quick reply to this message  
•
•
Join Date: Jun 2008
Posts: 849
Reputation: R0bb0b is on a distinguished road 
Solved Threads: 67
R0bb0b's Avatar
R0bb0b R0bb0b is offline Offline
Practically a Posting Shark

Re: Looking for PHP Web Spider Script

 
0
  #2
Sep 27th, 2008
•
•
•
•
Originally Posted by monsterpot View Post
Apologies first if this is covered elsewhere - I searched but could not find.

I am looking for a way to search the web for the presence of a JavaScript code snippet within the HTML <body> of a web page. I would specify the code snippet and send the bot on its way, it would come back with either a number of results, or a list of pages.

I realise there are billions of web pages so don't know whether this is feasilble or not.

The purpose is to determine the number of participating sites in a particular network. (Currently uncontrolled so any site could have the code on)

Any ideas on this?

Thanks
I don't think you will find too many php web spider scripts only because php is an interpreted language and will be quite slow. I would look into Java probably.
Last edited by R0bb0b; Sep 27th, 2008 at 4:37 pm.
“Be who you are and say what you feel because those who mind don't matter and those who matter don't mind.” - Dr. Seuss

-- The documentation is inevitable, you may get away with it for a little while but eventually you too will have to do the deed.
Reply With Quote Quick reply to this message  
•
•
Join Date: Sep 2008
Posts: 15
Reputation: heenix is an unknown quantity at this point 
Solved Threads: 1
heenix heenix is offline Offline
Newbie Poster

Re: Looking for PHP Web Spider Script

 
0
  #3
Sep 27th, 2008
Hi,

I agree with RobRob here, PHP just isn't powerful enough to be doing this sort of task. PHP is not a compiled language, so it is interpreted into machine-readable code on-the-fly. This is very slow in comparison to compiled languages such as C and C++ which naturally operate at a closer level to the hardware. Even if you had unlimited power and resources, PHP is very restrictive as it only runs once, and thanks to execution timeouts, you can't do very much.

If you're still interested in making a spider, I must say that it's no easy feat. You would be much better looking into C and C++ in subsidary forums, with most of you're focus spent on understanding networking.

Thanks,
Christopher Lord
Reply With Quote Quick reply to this message  
•
•
Join Date: Sep 2008
Posts: 202
Reputation: FlashCreations is an unknown quantity at this point 
Solved Threads: 17
FlashCreations's Avatar
FlashCreations FlashCreations is offline Offline
Posting Whiz in Training

Re: Looking for PHP Web Spider Script

 
0
  #4
Sep 28th, 2008
How about trying a Google search? You could search within the HTML for your Javascrpit code. If you would like it in script form, try using the Google Ajax Search widget and simply count the results!!!
Last edited by FlashCreations; Sep 28th, 2008 at 4:43 pm.
FlashCreations
(aka PhpMyCoder)

About Me | My Blog | Contact Me
Reply With Quote Quick reply to this message  
•
•
Join Date: Sep 2007
Posts: 1,546
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 137
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Posting Virtuoso

Re: Looking for PHP Web Spider Script

 
0
  #5
Sep 29th, 2008
•
•
•
•
•
•
•
•
I don't think you will find too many php web spider scripts only because php is an interpreted language and will be quite slow.
•
•
•
•
I agree with RobRob here, PHP just isn't powerful enough to be doing this sort of task.
Well from the quotes above, I may just be able to prove all of that wrong with a PHP bot series I am working on. This post is not advertiesment but is to make the users 'R0bb0b' and 'heenix' and 'monsterpot' aware of what PHP is really capable of.
I have managed to make a bot which will index all of the urls that a linked to (within a tree type format) and will keep on indexing the web until it runs out of links. I will provide you with the source for you to make your javascript finder and currently the below script only indexes urls which is needed to scan more than one page.
php Syntax (Toggle Plain Text) 
  1. if (isset($_GET['url']))
  2. 	{
  3. 	include("db.php");
  4. 	mysql_connect($dbhost,$accountname,$password)
  5. 	or die("Could not connect to MySQL server");
  6. 	mysql_select_db($database) or die(mysql_error()."Could not select database");
  7. 	$file=file_get_contents($_GET['url']);
  8. 	$links=preg_split('/(href\=\'|href\=\"|href\=)/is',$file);
  9. 	//$file=preg_replace('/(.*)(\ href\=\"|\ href\=\'|\ href=)(.*\ |.*\"|.*\'|.*>)/is','$2|^=^|',$file,1);
  10. 	//echo $file."<br>";
  11.  
  12. 	mysql_query("INSERT INTO `indextemp` SET `url`='".$_GET['url']."', `stage`='1'");
  13. 	$id=1;
  14. 	while (isset($links[$id]))
  15. 		{
  16. 		$links[$id]=preg_replace("/([^\'])\'(.*)/is",'$1',$links[$id]);
  17. 		$links[$id]=preg_replace("/([^\"])\"(.*)/is",'$1',$links[$id]);
  18. 		$links[$id]=preg_replace("/([^\>])\>(.*)/is",'$1',$links[$id]);
  19. 		$links[$id]=preg_replace("/([^ ])\ (.*)/is",'$1',$links[$id]);
  20. 		$links[$id]=preg_replace("/([^\'])\'(.*)/i",'$1',$links[$id]);
  21. 		$links[$id]=preg_replace("/([^\"])\"(.*)/i",'$1',$links[$id]);
  22. 		$links[$id]=preg_replace("/([^\>])\>(.*)/i",'$1',$links[$id]);
  23. 		$links[$id]=@preg_replace("/([^ ])\ (.*)/i",'$1',$links[$id]);
  24.  
  25. 		$ifexists=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  26. 		if (mysql_num_rows($ifexists)==0 && strlen($links[$id])>16)
  27. 			{
  28. 			mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  29. 			echo $links[$id]."<br>";
  30. 			}
  31. 		$id+=1;
  32. 		}
  33. 	unset ($links);
  34. 	$continue=1;
  35. 	while ($continue=1)
  36. 		{
  37.  
  38. 		$sqllinksa=mysql_query("SELECT * FROM `indextemp` WHERE `stage`='0'");
  39. 		while ($sqllinks=mysql_fetch_array($sqllinksa))
  40. 			{
  41. 			$file=file_get_contents($sqllinks['url']);
  42. 			$links=preg_split('/(href\=\'|href\=\"|href\=)/is',$file);
  43.  
  44. 			mysql_query("UPDATE `indextemp` SET `stage`='1' WHERE `url`='".$sqllinks['url']."'");
  45. 			$id=1;
  46. 			while (isset($links[$id]))
  47. 				{
  48. 				$links[$id]=preg_replace("/([^\'])\'(.*)/is",'$1',$links[$id]);
  49. 				$links[$id]=preg_replace("/([^\"])\"(.*)/is",'$1',$links[$id]);
  50. 				$links[$id]=preg_replace("/([^\>])\>(.*)/is",'$1',$links[$id]);
  51. 				$links[$id]=preg_replace("/([^ ])\ (.*)/is",'$1',$links[$id]);
  52. 				$links[$id]=preg_replace("/([^\'])\'(.*)/i",'$1',$links[$id]);
  53. 				$links[$id]=preg_replace("/([^\"])\"(.*)/i",'$1',$links[$id]);
  54. 				$links[$id]=preg_replace("/([^\>])\>(.*)/i",'$1',$links[$id]);
  55. 				$links[$id]=preg_replace("/([^ ])\ (.*)/i",'$1',$links[$id]);
  56.  
  57. 				$ifexist=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  58. 				if (strlen($links[$id])>5 && mysql_num_rows($ifexist)==0)
  59. 					{
  60. 					mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  61. 					echo $links[$id]."<br>";
  62. 					} else { unset($ifexists['url']); }
  63. 				$id+=1;
  64. 				}
  65.  
  66.  
  67. 			$ifexists=mysql_query("SELECT * FROM `indextemp` WHERE `url`='".$links[$id]."'");
  68. 			$ifexists=mysql_fetch_array($ifexists);
  69. 			if (!isset($ifexists['url']) && strlen($links[$id])>5)
  70. 				{
  71. 				mysql_query("INSERT INTO `indextemp` SET `url`='".$links[$id]."', `stage`='0'");
  72. 				echo $links[$id]."<br>";
  73. 				}
  74. 			}
  75. 		$checkcontinue=mysql_query("SELECT * FROM `indextemp` WHERE `stage`='0'");
  76. 		if (mysql_num_rows($checkcontinue)==0)
  77. 			{
  78. 			$continue=0;
  79. 			break;
  80. 			}
  81. 		}
  82. 		//
  83. 	}
  84. echo "<form><input type='text' name='url' size=50><input type='submit' value='index'></form>";
  85. ?>

Below is a second file named 'db.php' with sql configurations.
php Syntax (Toggle Plain Text) 
  1. <?
  2. $accountname='root';
  3. $password='';
  4. $dbhost='localhost';
  5. $database='mydatabasename';
  6. ?>
Also, in the mysql database, there is a table named 'tempindex' with the columns 'url' and 'stage'. (but remove all the quotes in the previous sentence.)

Comment to monsterpot:
If you are still interested in making a PHP bot to find your javascript on the web then just let me know as I can help you there. Also I would need to know exactly what that javascript code is and what parts of that javascript code can change.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - Oopy Doopy Do 2U2!
Reply With Quote Quick reply to this message  
•
•
Join Date: Jun 2008
Posts: 849
Reputation: R0bb0b is on a distinguished road 
Solved Threads: 67
R0bb0b's Avatar
R0bb0b R0bb0b is offline Offline
Practically a Posting Shark

Re: Looking for PHP Web Spider Script

 
0
  #6
Sep 29th, 2008
•
•
•
•
Originally Posted by cwarn23 View Post
Well from the quotes above, I may just be able to prove all of that wrong with a PHP bot series I am working on. This post is not advertiesment but is to make the users 'R0bb0b' and 'heenix' and 'monsterpot' aware of what PHP is really capable of.
I have managed to make a bot which will index all of the urls that a linked to (within a tree type format) and will keep on indexing the web until it runs out of links. I will provide you with the source for you to make your javascript finder and currently the below script only indexes urls which is needed to scan more than one page.
Sure, nobody said it couldn't be done, it'll just take all day, possibly all week depending on how many sites you plan to spider. Google actually claims to index 2e9 pages. Let's say you get a fourth that at some point(500,000,000), and let me be generous and say that each page has an average of 10 links per page. If I am reading your script right that would be 5,000,000,000 loops. Last time I clocked PHP it was running about 600000 loops per minute on a 3g processor with 1g ram. That would average just over 833 minutes given that you had absolutely no memory leaks while the script is running and this doesn't include the file_get_contents() and mysql_query() functions that you will be running. I personally wouldn't do it with php because PHP is not a multi-threading language(which would be extremely helpful in this case) and PHP is several times slower than C++.

But for a side project, maybe.
Last edited by R0bb0b; Sep 29th, 2008 at 1:34 am.
“Be who you are and say what you feel because those who mind don't matter and those who matter don't mind.” - Dr. Seuss

-- The documentation is inevitable, you may get away with it for a little while but eventually you too will have to do the deed.
Reply With Quote Quick reply to this message  
•
•
Join Date: Sep 2007
Posts: 1,546
Reputation: cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about cwarn23 has a spectacular aura about 
Solved Threads: 137
cwarn23's Avatar
cwarn23 cwarn23 is offline Offline
Posting Virtuoso

Re: Looking for PHP Web Spider Script

 
0
  #7
Sep 29th, 2008
•
•
•
•
Sure, nobody said it couldn't be done, it'll just take all day, possibly all week depending on how many sites you plan to spider. Google actually claims to index 2e9 pages.
Just to add to the statistics, I remember reading in Google 'to the power of project' that Google now claims to have 1 trillion pages indexed. Also from the Google web alerts I have
(eg. inurl:cwarn23.info) I have found that Google usually revisits the same website within 8 days but one time in the past year has taken 14 days. Also I have tested my script to index an average of 16 pages per second (at the most). So I hope you find those statistics useful.
Try not to bump 10 year old threads as it can be really annoying.
http://syntax.cwarn23.net/
Smilies: ^_* +_+ v_v -_- *~*`
My favourite PC. - Oopy Doopy Do 2U2!
Reply With Quote Quick reply to this message  
•
•
Join Date: Dec 2008
Posts: 2
Reputation: Mantriur is an unknown quantity at this point 
Solved Threads: 0
Mantriur Mantriur is offline Offline
Newbie Poster

Re: Looking for PHP Web Spider Script

 
0
  #8
Dec 30th, 2008
The stupidity in this thread made me register an account. Originally I was googling for the same thing the original poster is looking for.

When you crawl the web, you will spend most time waiting for network packets and saving the data some place. A spider is a perfect example of a piece of code, where execution time does not matter at all. You could write it in commodore basic 2.0 and wouldn't notice a difference.

Creating a spider in an unsuitable language like C++ will double your development effort for an actual performance gain in the first percentile.

The execution time limit in PHP is actually configurable. (Doh). It's usually disabled for command line execution. PHP only runs once? What happens then? The script self destructs?

Recommending java over PHP for performance reasons only makes sense if you are religious and worship The Java.

Naturally running a web crawler is not a task for days or even weeks. It's closer to years. If the op was looking to crawl a single site or two, he'd probably use one of the perfectly fine windows client applications and not look for a script.

There are opcode caches for PHP, which make it a just-in-time compiled language.

Assuming that there are no memory leaks is quite generous. Can we also assume that the world is round? When you "clock" a programming language, it would be kinda helpful to know what that loop was running, which operating system you were using, the bus width, and the compiler flags for the executable. The amount of memory does rather not matter.

Oh, and ...

PHP Syntax (Toggle Plain Text) 
  1. <?php
  2. $count = 0;
  3. $now = microtime(true);
  4. while ( ($now+1) > microtime(true)) $count++;
  5. print "Loops per second: ".number_format($count)."\n";
  6. ?>
  7.  
  8. workhorse:~# php loop.php
  9. Loops per second: 2,222,026

... you are full of it.

But let's assume you actually benchmarked the script in question. Let's also assume an average text weight of 50kb for a web page. Then your 3g processor (mobile phone?) could spider 30 gigabyte per minute. That's ~500 megabytes per second. Phat subsystem there. Mysql cluster with memory tables on 10GbE?

There are simple ways to split the websites to crawl between several instances of the script. You do not need threads. You can multi-task.

You remind me of that dude who threatened to "hack my website" and backed that claim with a traceroute. Please stop giving technical advice. Thank you.
Reply With Quote Quick reply to this message  
•
•
Join Date: Dec 2008
Posts: 2
Reputation: Mantriur is an unknown quantity at this point 
Solved Threads: 0
Mantriur Mantriur is offline Offline
Newbie Poster

Re: Looking for PHP Web Spider Script

 
0
  #9
Dec 30th, 2008
But to answer the original question:

http://vision-media.ca/resources/php...aper-5-minutes

Just modify one of the regexp to identify the script tags you are looking for and use the get_links function to identify the next target.

Dump the links into a mysql table together with a flag if/when crawled and feed the spider from that table. Dump positives into another table with the URL. You can then query the results from that one any time.

If you run this on an external server or webspace, observe your bandwidth/volume limits. The internet is big.
Last edited by Mantriur; Dec 30th, 2008 at 9:33 am.
Reply With Quote Quick reply to this message  
•
•
Join Date: Jul 2009
Posts: 28
Reputation: crohole is an unknown quantity at this point 
Solved Threads: 0
crohole crohole is offline Offline
Light Poster

Re: Looking for PHP Web Spider Script

 
0
  #10
Jul 14th, 2009
I had try the script..did it just crawl in homepage..And it crawl external link to like google and ect..

How to make it just crawl the internal link in all pages, and it never crawl the same internal link..Please tell me how to do that..? Urgent
Reply With Quote Quick reply to this message  
Reply
1 2

Quick Reply to Thread
This thread is more than three months old.
Perhaps start a new thread instead?
•
•
•
•
•
•
•
•
Message:
•
•
•
•



Similar Threads
Other Threads in the PHP Forum


Views: 4677 | Replies: 11
Thread Tools Search this Thread



Tag cloud for PHP
.htaccess access ajax apache api array beginner binary broken cakephp checkbox class clean cms code countingeverycharactersfromastring cron curl database date directory display download dynamic echo email error file files folder form forms function functions google href htaccess html image include insert integration ip java javascript joomla jquery limit link login loop mail match menu methods mlm mod_rewrite multiple mysql oop parse paypal pdf php problem query radio random recursion regex remote script search select server sessions simple sms soap source space spam speed sql structure syntax system table tutorial update updates upload url validation validator variable video web xml youtube
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC