Member Avatar for Allison2009

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

  • 10 Contributors
  • 11 Replies
  • 200 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by skseo
orangeversion commented: Learned something new today +0

Dani AI

Generated

Good first steps from and — adding a robots.txt is the right, low-risk starting point. Robots.txt tells well‑behaved crawlers what to avoid, but it only controls crawling (not whether a URL can appear in an index) and obedient behavior varies by bot. For that reason, check indexing implications before deciding which approach to use. (developers.google.com)

If the hits truly are from Yandex, use Yandex.Webmaster first — it exposes a site “crawl rate” control so you can throttle how often Yandex’s robots request pages (this replaced reliance on a crawl‑delay token). If you need to reduce load quickly, Yandex also recommends cleaning up robots.txt rules so the bot isn’t wasting cycles on duplicates or service pages. (yandex.com)

Confirm the visitor is a genuine Yandex robot before blocking. Pull the IPs from your server logs, do a reverse DNS on each IP and then forward lookup to match the address — valid Yandex hosts end with yandex.ru, yandex.net or yandex.com. That prevents accidentally blocking legitimate search engines or helpful services. Example commands:

host 1.2.3.4
dig -x 1.2.3.4 +short
nslookup 1.2.3.4

Follow the reverse/forward lookup procedure in Yandex’s guide. (yandex.com)

If a robot ignores robots.txt or is spoofing, enforce at the server or firewall level. Apache (2.4+) can deny an IP or block by user agent via access rules; Nginx can return 403/444 for matched user agents or IPs. Test changes on a staging host, then monitor logs to confirm the load drops. Example patterns (adapt to your stack and IP list):

# Apache 2.4 (in vhost or .htaccess)
<RequireAll>
  Require all granted
  Require not ip 203.0.113.0/24
</RequireAll>

# Nginx (server block)
if ($http_user_agent ~* "Yandex") { return 403; }

See Apache and Nginx docs for safe syntax and caveats. (httpd.apache.org)

Summary: robots.txt first, verify bot identity, use Yandex.Webmaster to throttle Yandex, and move to server/firewall blocks only if the crawler is non‑compliant. Keep monitoring logs and be mindful that blocking Yandex will stop Yandex indexing/traffic for the site. (yandex.com)

Recommended Answers

All 11 Replies

Member Avatar for inspiroHost

Create a robots.txt file with the following content. This will prevent Yandex from crawling your site.

User-agent: Yandex
Disallow: /

User-agent: Yandex/1.01.001
Disallow: /

Member Avatar for dailyearner

edit your robots.txt file

Member Avatar for Allison2009

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

Member Avatar for julibe

Very useful information, thanks.

Member Avatar for inspiroHost

Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.

Thanks again

No problem at all, glad to help :)

Member Avatar for pmolds

write your robots.txt file and stop it.

Member Avatar for ameto

You can also do it on per page basis..put meta robot tag in head section of your page...

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

Member Avatar for vicjg

Is there a drawback to restricting Yandex? i.e. Is anyone seeing quality traffic from the site?

Edited by vicjg because: n/a
Member Avatar for joelchrist

you can use robot.txt file to stop crawling particular pages.

Member Avatar for AirForceOne

You could block them in your robots.txt file if such crawlers obey the rules. If not, I would recommend you to block them directly via IP blocker inside your hosting control panel.

Have a nice day,

Member Avatar for skseo

Hi,

My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.

Any advice please.

Thanks in advance

nice! your site is continuly indexing.what strategy you are using ? good knowledge i got here.

thanks

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.