Member Avatar for Allison2009

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

  • 10 Contributors
  • 11 Replies
  • 185 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by skseo
orangeversion commented: Learned something new today +0

Recommended Answers

All 11 Replies

Member Avatar for inspiroHost

Create a robots.txt file with the following content. This will prevent Yandex from crawling your site.

User-agent: Yandex
Disallow: /

User-agent: Yandex/1.01.001
Disallow: /

Member Avatar for dailyearner

edit your robots.txt file

Member Avatar for Allison2009

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

Member Avatar for julibe

Very useful information, thanks.

Member Avatar for inspiroHost

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

No problem at all, glad to help :)

Member Avatar for pmolds

write your robots.txt file and stop it.

Member Avatar for ameto

You can also do it on per page basis..put meta robot tag in head section of your page...

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

Member Avatar for vicjg

Is there a drawback to restricting Yandex? i.e. Is anyone seeing quality traffic from the site?

Edited by vicjg because: n/a
Member Avatar for joelchrist

you can use robot.txt file to stop crawling particular pages.

Member Avatar for AirForceOne

You could block them in your robots.txt file if such crawlers obey the rules. If not, I would recommend you to block them directly via IP blocker inside your hosting control panel.

Have a nice day,

Member Avatar for skseo

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

nice! your site is continuly indexing.what strategy you are using ? good knowledge i got here.

thanks

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.