Member Avatar for Dani

Is it no longer possible to change the crawl rate for a site in Google Webmaster Tools? Can't find the option anymore.

  • 3 Contributors
  • 3 Replies
  • 100 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by ellena980

Dani AI

Generated

Quick context tied to the replies: asked about the crawl-rate control, pointed to Google's help, and mentioned the slider. That slider existed in the older Webmaster Tools UI, but many sites do not see a manual crawl-rate control in the newer Search Console. If you can’t find the option, here are practical, reliable alternatives and what to check.

How to diagnose and reduce unwanted crawl load:

  • Use Search Console's Crawl Stats and your server logs to see which URLs Googlebot requests, response codes, and timing. Look for repeated hits to low-value paths (search results, faceted navigation, session IDs, calendar feeds).
  • Reduce surface area: disallow low-value directories in robots.txt, consolidate duplicates with canonical tags, clean up URL-parameters or use the parameter tool, and submit focused sitemaps so Google prioritizes important pages.
  • Improve server handling: add caching, faster pages, or a CDN so crawls are less costly. Googlebot ignores the robots.txt "Crawl-delay" directive, so do not rely on it.
  • If you need an immediate throttle, use server-level rate limiting or temporary 503 responses (these signal temporary unavailability). Example Nginx approach:
# limit requests per IP to ~1req/sec, burst up to 5
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

server {
    ...
    location / {
        limit_req zone=one burst=5 nodelay;
        ...
    }
}

Cautions and next steps:

  • Over-restricting crawl slows discovery of new content and can harm indexing. Disallowing via robots.txt prevents crawling (and may leave URLs indexed without content); if you need pages removed, prefer allowing crawl + using noindex.
  • Capture server logs (timestamps, user-agent, response codes) before making changes so you can measure impact. If crawl traffic still looks abnormal after tuning, verify property type/interface in Search Console and consider contacting your host or Google Search Central resources.

Recommended Answers

All 3 Replies

Member Avatar for Member #949455

Is it no longer possible to change the crawl rate for a site in Google Webmaster Tools? Can't find the option anymore.

I think it's still there:

Member Avatar for Dani

Oh there it is! I couldn't find it before for some reason. Thanks! :)

Member Avatar for ellena980

In webmaster tool you will able to find just check that and slide the slider to conrol the rate.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.