0

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

10
Contributors
11
Replies
15
Views
7 Years
Discussion Span
Last Post by skseo
0

Create a robots.txt file with the following content. This will prevent Yandex from crawling your site.

User-agent: Yandex
Disallow: /

User-agent: Yandex/1.01.001
Disallow: /

0

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

0

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

No problem at all, glad to help :)

0

You can also do it on per page basis..put meta robot tag in head section of your page...

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

0

Is there a drawback to restricting Yandex? i.e. Is anyone seeing quality traffic from the site?

Edited by vicjg: n/a

0

You could block them in your robots.txt file if such crawlers obey the rules. If not, I would recommend you to block them directly via IP blocker inside your hosting control panel.

Have a nice day,

0

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

nice! your site is continuly indexing.what strategy you are using ? good knowledge i got here.

thanks

This question has already been answered. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.