0

Hey.

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

Can this be done?

Thanks.

5
Contributors
4
Replies
6
Views
5 Years
Discussion Span
Last Post by Igal-Incapsula
0

@louie540

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

This question is more suited for Internet Marketing Section rather than PHP section. I understand what you are mention about the robot.txt but if you have a code in PHP than it is OK to post it here. Otherwise post it there. If you have a PHP code just post the code in the PHP section.

0

Hi
Googlebot will ALWAYS comply with robots.txt directives.

I think you may be suffering from Googlebot impersonation attempt.

Here is a link to learn more about that:
http://www.incapsula.com/the-incapsula-blog/item/369-was-that-really-a-google-bot-crawling-my-site

My advise is:

  • Use Google WMT and do a manual "Fetch" for the button URL. If you succesed than your robots.txt is not really blocking it and you'll need to modify it accordingly.

  • If "Fetch" fails (meaning your robots.txt settings are fine) than you should get the user-agent and IP info of this "Googlebot" and verify them on http://www.Botopedia.org

If user-agent data or IPs won't match you'll know that something is wrong. In this case the visiting IP should be blocked because this is either a spam attempt or a vuln. scan performed for future hacking purposes. (somethinkg like Havij for exmple)

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.