Display content based on IP

[louie540]

Hey.

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

Can this be done?

Thanks.

[LastMitch]

@louie540

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

This question is more suited for Internet Marketing Section rather than PHP section. I understand what you are mention about the robot.txt but if you have a code in PHP than it is OK to post it here. Otherwise post it there. If you have a PHP code just post the code in the PHP section.

[leviathan185]

Google will pry into your life not matter how much you try and stop them.... Just sayin

[Echo89]

You could achieve this with the user agent maybe?

[Igal-Incapsula]

Hi
Googlebot will ALWAYS comply with robots.txt directives.

I think you may be suffering from Googlebot impersonation attempt.

Here is a link to learn more about that:
http://www.incapsula.com/the-incapsula-blog/item/369-was-that-really-a-google-bot-crawling-my-site

My advise is:

• Use Google WMT and do a manual "Fetch" for the button URL. If you succesed than your robots.txt is not really blocking it and you'll need to modify it accordingly.

• If "Fetch" fails (meaning your robots.txt settings are fine) than you should get the user-agent and IP info of this "Googlebot" and verify them on http://www.Botopedia.org

If user-agent data or IPs won't match you'll know that something is wrong. In this case the visiting IP should be blocked because this is either a spam attempt or a vuln. scan performed for future hacking purposes. (somethinkg like Havij for exmple)