Hello,
Bandwidth on one of our sites was severely messed with on the 28th of this month. The cpanel only tracks daily access logs and didnt archive them(it does now), using aw stats I found our bot traffic to be as follows:
Googlebot 145686+43 7.47 GB 28 Jul 2010 - 10:18
Unknown robot (identified by 'bot*') 91541+417 4.78 GB 28 Jul 2010 - 07:12
I have blocked bot* using htaccess:
RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]
I have been informed that this can interfere with traffic, what should I do? should I wait for it to happen again then check logs for IP/Agent name or continue to block unkown robots?
I did a dns lookup on the googlebot entries I do have and they check out.
