How to stop a crawler from crawling my website

[Allison2009]

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

[inspiroHost]

Create a robots.txt file with the following content. This will prevent Yandex from crawling your site.

User-agent: Yandex
Disallow: /

User-agent: Yandex/1.01.001
Disallow: /

[dailyearner]

edit your robots.txt file

[Allison2009]

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

[julibe]

Very useful information, thanks.

[inspiroHost]

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

No problem at all, glad to help :)

[pmolds]

write your robots.txt file and stop it.

[ameto]

You can also do it on per page basis..put meta robot tag in head section of your page...

[vicjg]

Is there a drawback to restricting Yandex? i.e. Is anyone seeing quality traffic from the site?

[joelchrist]

you can use robot.txt file to stop crawling particular pages.

[AirForceOne]

You could block them in your robots.txt file if such crawlers obey the rules. If not, I would recommend you to block them directly via IP blocker inside your hosting control panel.

Have a nice day,

[skseo]

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

nice! your site is continuly indexing.what strategy you are using ? good knowledge i got here.

thanks