Member Avatar for JayGeePee

I found a crawl detection script on the internet that works great. But at the moment I only have Googlebot. I'm trying to add YahoosLurp, and Ask Jeeves, and any other popular web crawler. PHP is definetly my weakness

Heres the script

<?php

$useragent = $_SERVER["HTTP_USER_AGENT"];

if (stripos($useragent,"Googlebot"))
{
 //Found! Create file
 $file = fopen("crawled.txt","a");
 fwrite($file,"You've been crawled\n");
}

?>

Its saved in a text file called crawled.txt

Im trying to add extra crawlers to the script, with the date.

Edited by JayGeePee because: Didn't add code brackets
  • 2 Contributors
  • 1 Reply
  • 178 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by mbhanley
Member Avatar for mbhanley

Here you go use this and then include the text files into a textarea it works pretty sweet. Hope this helps.

<?php

/* 
Search engine spiders
Yahoo! Slurp
Msnbot
Googlebot
Ask Jeeves
*/

$useragent = $_SERVER["HTTP_USER_AGENT"];
$ref = $_SERVER['HTTP_REFERER'];
$host = $_SERVER['HTTP_HOST'];
$ip = $_SERVER['REMOTE_ADDR'];
$date = date('d/m/Y H:i:s');


if (strpos($useragent,"Firefox"))
{
	//found! create file
	$file = fopen("./crawldetector/Visitor.txt","a");
	fwrite($file,""."\nVisitor Stat: \n"."UserAgent: $useragent"."\n"."Date & Time: $date"."$ref \n"."$ip\n"."$host"."\n ----------------------- \n"."");
}


if (strpos($useragent,"Yahoo! Slurp"))
{
	//found! create file
	$file = fopen("./crawldetector/Yahoo.txt","a");
	fwrite($file,""."\nCrawl Stat: \n"."UserAgent: $useragent"."\n"."Date & Time: $date"."$ref \n"."$ip\n"."$host"."\n ----------------------- \n"."");
}

if (strpos($useragent,"Googlebot"))
{
	//found! create file
	$file = fopen("./crawldetector/Googlebot.txt","a");
	fwrite($file,""."\nCrawl Stat: \n"."UserAgent: $useragent"."\n"."Date & Time: $date"."$ref \n"."$ip\n"."$host"."\n ----------------------- \n"."");
}

if (strpos($useragent,"Msnbot"))
{
	//found! create file
	$file = fopen("./crawldetector/Msnbot.txt","a");
	fwrite($file,""."\nCrawl Stat: \n"."UserAgent: $useragent"."\n"."Date & Time: $date"."$ref \n"."$ip\n"."$host"."\n ----------------------- \n"."");
}

if (strpos($useragent,"Ask Jeeves"))
{
	//found! create file
	$file = fopen("./crawldetector/AskJeeves.txt","a");
	fwrite($file,""."\nCrawl Stat: \n"."UserAgent: $useragent"."\n"."Date & Time: $date"."$ref \n"."$ip\n"."$host"."\n ----------------------- \n"."");
}


?>

Textarea's with the text files to include.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Crawler Stats</title>
</head>

<body>

<h4>Visitors</h4>
<textarea rows="10" cols="70">
<?php include('visitor.txt'); ?>
</textarea>

<br />
<h4>Google</h4>
<textarea rows="10" cols="70">
<?php include('googlebot.txt'); ?>
</textarea>

<br />
<h4>Yahoo</h4>
<textarea rows="10" cols="70">
<?php include('Yahoo.txt'); ?>
</textarea>

<br />
<h4>Bing</h4>
<textarea rows="10" cols="70">
<?php include('Msnbot.txt'); ?>
</textarea>

<br />
<h4>Ask Jeeves</h4>
<textarea rows="10" cols="70">
<?php include('AskJeeves.txt'); ?>
</textarea>

</body>
</html>
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.